CROSS-REFERENCE TO RELATED APPLICATION

This application claims priority to U.S. Provisional Patent App. No. 62/944,678, filed on December 6, 2019, which is incorporated herein by reference in its entirety.

TECHNICAL FIELD

This disclosure relates to a system and method for optical 3D measurement that combines elements of Time-of-Flight (TOF) sensing as well as elements of Active-Triangulation (AT) sensing. BACKGROUND

Three-dimensional optical depth sensing is a widely used technique across a multitude of fields including automation, human-machine interaction, and security. The purpose of depth sensing is to collect three-dimensional coordinates of objects within the field of view of a depth capture system, such that these measurements may be used for calculation or display. For example, mobile robots commonly use depth sensors to detect their surroundings and plan safe travel paths.

A variety of depth sensor modalities exist, including active-triangulation and time-of-flight. Active-triangulation systems generally include a projector and a camera. The projector emits a known pattern of illumination into a field of view. A camera is separated from the projector by a baseline distance and captures the reflection of the pattern from objects in the field of view. The captured pattern may be distorted due to the surface geometry of the reflecting objects. The camera image is processed to determine the geometric shape of the surface.

Time-of-flight systems also generally include a projector and camera. The projector emits a pulse or pulses of light at a known time. This light reflects off objects in the field of view of the system. A portion of the reflected light arrives at the sensor, which measures the arrival time of the light. Knowing the speed of light, the round-trip distance of the pulse or pulses can be found, and the surface geometry that reflected the light can be determined.

Because each of these techniques offer different performance trade-offs, various techniques have been described that extract both active-triangulation and time-of-flight information. Some such systems switch between these two modes of operation using different projectors or sensors under different conditions. Other systems overlay structure information on a time-of-flight system so that two separate measurements may be made.

In view of the drawbacks of current systems, there is a need for a new depth capture method that does not rely on a time of flight measurement or an active-triangulation measurement, but rather combines aspects of both modalities to produce a single depth estimate of greater worth (e.g., accuracy, speed of determination) with less uncertainty than has previously been achieved.

The methods and system described herein enable a novel technique for combined-phase depth sensing. This technique provides high resolution and lower noise than equivalent active-triangulation or time-of-flight systems at various ranges. This technique also allows for phase unwrapping and autocalibration techniques. SUMMARY

In an embodiment, a system includes a projector configured to project a wave of light; a sensor configured to sense a reflection or scattering of the wave of light; and a computing device configured to calculate a measurement based on the sensed reflection or scattering.

In an embodiment, a method includes projecting, by a projector, a wave of light; sensing, by a sensor, a reflection or scattering of the wave of light; and calculating, by a processor, a measurement based on the sensed reflection or scattering.

The foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the claims.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings, which are incorporated in and constitute a part of this disclosure, illustrate various disclosed embodiments. In the drawings:

Fig. 1 is a conceptual diagram of an exemplary depth capture system.

Fig. 2 is a group of exemplary plots demonstrating the operation of the ‘fast-sweep’ method.

Fig. 3 is a conceptual diagram of exemplary depth capture system geometry.

Fig. 4 is a flow-chart of an exemplary depth capture process.

Fig. 5 is a plot comparing exemplary predicted system performance.

Fig. 6 depicts an exemplary projector for depth capture.

Fig. 7 depicts an exemplary projector for depth capture.

Fig. 8 illustrates exemplary details of a projector embodiment.

Fig. 9 is a group of exemplary plots demonstrating the operation of the ‘slow-sweep method.

Fig. 10 depicts an exemplary projector for depth capture.

Fig. 11 depicts an exemplary a projector for depth capture.

Fig. 12 depicts an exemplary projector for depth capture.

DETAIFED DESCRIPTION

The following detailed description refers to the accompanying drawings. Wherever possible, the same reference numbers are used in the drawings and the following description to refer to the same or similar parts. While several illustrative embodiments are described herein, modifications, adaptations and other implementations are possible. For example, substitutions, additions or modifications may be made to the components illustrated in the drawings, and the illustrative methods described herein may be modified by substituting, reordering, removing, or adding steps to the disclosed methods. Accordingly, the following detailed description is not limited to the disclosed embodiments and examples. Instead, the proper scope is defined by the appended claims.

Described herein is a method, system, and embodiments of a 3D depth sensor. An embodiment is shown in Fig. 1. In some embodiments, a dynamic pattern projector (115), which may be configured to project (e.g., emit) a wave of light, and a time-of-flight sensor (100), which may be configured to sense a reflection or scattering of a wave of light (e.g., projected by projector 115), that are offset from each other by a baseline distance (135) are provided. A wave of light may also be referred to as a wave, light, or a light wave. The dynamic pattern projector (115) may have optical axis (150), based on its orientation that may be different from an optical axis (145) of the time-of-flight sensor (100), which may have its own orientation. The projector and sensor have overlapping fields of view (105,120). The projector may output a time and space-varying field of light, as described below. An example ray of such emitted light (125) may intersect with an object (130) within a field of view. Scattered or reflected light (110) may then be received by the sensor (100). A computing device may process the signal from the sensor, such that based on the measured properties of this light, a depth estimate is provided in three-dimensional space for pixels of the sensor that receive light having emanated from the projector. This process is described sequentially in the flow chart Fig. 4. A computing device may include at least one device configured to execute instructions to carry out any of the steps described herein (e.g., in some embodiments multiple devices, which may constitute a computation system, may be used).

As can be appreciated from the described embodiments, a computing device may be configured to calculate a measurement by modeling a sensed reflection or scattering (e.g., of a wave of light output by a projector). For example, the computing device may model at least a portion of the sensed reflection or scattering as a function (e.g., according to the equations and techniques described below).

One embodiment disclosed herein is are referred to as the ‘slow-sweep’ method and another embodiment is referred to as the ‘fast-sweep’ method. Both methods can be implemented using the system described above, wherein a projector outputs patterns that vary in both space and time such that after reflection and sampling by a time-of-flight sensor, the measured phase from the sensor may be converted to a depth value. Because the phase of the projector is coupled in space and time, this method combines aspects of active-triangulation and time-of-flight, but is distinguished over both methods and from other methods that overlay or interleave time-of-flight and active-triangulation measurements, as will be described further below. Also disclosed are several possible projector embodiments that can be used, such as in combination with additional hardware and software components for implementing the slow-sweep and fast-sweep methods, to recalibrate the system.

Focusing now on the ‘fast-sweep’ method according to embodiments of the present disclosure. The ‘fast-sweep’ method includes a periodic pattern that sweeps in the direction of periodicity at a fast rate. The term “fast rate” is used herein in the context of time-of-flight modulation. In one example, a periodic pattern may sweep in the direction of periodicity at a rate that may be above 1 MHz, and the pattern may be synchronous with a demodulation rate of a time-of-flight sensor. This sweep may also be continuous (e.g., a projector may project a wave of light continuously). In a further example, the sweep may be implemented in very small discrete increments, so as to approach continuity (e.g., a projector may project a wave of light in discrete increments). The direction of periodicity has a component that may be parallel to the baseline distance separating the projector and sensor. This projection may thereby illuminate any object in the field of view of the projector with intensity that may vary both in time and in space. Exemplary cross sections of this projection are demonstrated in Fig. 2.

In the example of Fig 2, the periodic component of the projected pattern is sinusoidal. The geometry of the system is shown in Fig 1, including the projector (115), angle Q (140) of an exemplary emitted ray (125) across the projector field of view (120), back reflected (110) by object (130), incident on the time-of-flight sensor (100) separated from the projector by baseline (135). In some embodiments, depth ‘Z’ is defined as the length of ray (110). As can be seen in an exemplary waveform (200), at a fixed value of depth Z and time t, the pattern is sinusoidally periodic across the projection angle Q. With appropriate demodulation, this projection can be considered to have phase f as seen in an exemplary waveform (205). To achieve the results shown, the pattern may be swept smoothly and rapidly in time, such as at frequencies above 1MHz. If a measurement is taken at both a fixed angle Q and a fixed depth Z, the intensity may be periodic in time, in this case sinusoidal, as is seen in an exemplary waveform (210). As with the spatial distribution of light, the temporal distribution of light can be considered to have a phase f, illustrated by an exemplary waveform (215). Given this distribution of light in angle Q and time t, an instantaneous measurement of light intensity along depth (Z) may also be approximately periodic. As shown in waveform 220, for any given projector ray (125) and at any given instant t, the light intensity along the depth (Z) of the projector ray may be approximately periodic. As mentioned above, the instantaneous measurement of light intensity along depth Z can be periodic or approximately periodic. The radial nature of the system will produce an expected shift in the frequency of the projection as a function of depth (Z) which can be accounted for by a computing device. Likewise, a projector may produce non-idealities that deviate from ideal periodicity, which may likewise be accounted for in subsequent computation. In some embodiments, a computing device may determine and/or demodulate a function. Demodulated, this function may have a phase value f, as shown in an exemplary waveform (225). In some embodiments, this demodulated function may contain more than one period, and a computing device may unwrap the demodulated function.

In some embodiments, such as after demodulating a function, a computing device may unwrap a function (e.g., the demodulated function). Unwrapping may comprise a process through which discontinuities in the phase function are removed to produce a continuous curve relating phase (f) to depth (Z). After unwrapping the phase-to-depth curve, a single relationship may exist between phase f and depth. Unwrapping is described in greater detail below.

In some embodiments, a computing device may determine a depth for a measured ray from the time-of-flight camera that captures reflected light originated by the projector, such as by using calibration parameters. In some embodiments, calibration parameters may be set by a user, determined during a calibration stage of use of the projector and time-of-flight camera, or a combination of both. After establishing the phase value for time-of-flight pixels, a computing device (e.g., computing device) may generate a depth image, which may be accomplished through a process termed ‘reconstruction’ . This process may consist of converting measured parameters to depth using an algorithm, which may be predicated on previously established calibration values.

To further illustrate this theory of combined active-triangulation and/or time-of-flight measurement, Fig. 3 illustrates an exemplary embodiment in which a projected spatial pattern is a sinusoid that varies periodically over an azimuthal polar emission angle Q (300). In this case the polar emission angle Q may increase in the clockwise direction with a value of 0 along a baseline segment (305). The sinusoidal projected intensity may be considered to have some constant positive offset value, as illumination may not be classified as negative. The baseline segment (305) has length b and may be considered to lie on the x-axis of the coordinate system. The depth Z may be orthogonal to this segment along the principal ray (310) of a camera. In some embodiments, a model may exist for the camera such that a single detection location over the camera sensor image plane receives light from a real world scene from along a ray that makes an angle b (315) with respect to a z-axis through the center of the camera. In some embodiments, b may be limited to horizontal azimuthal directions (x-z plane), though the calculation may be extended to all degrees of freedom in a 3d space. In some embodiments, another spatial pattern (330) may be projected from or toward the baseline segment (305), and may have a field of view (335) extending from one endpoint (320) to another (325).

For a given camera ray at angle b, the unambiguous, or unwrapped, phase (f _A t) that would be observed at the camera for a fixed (non-sweeping) projected pattern may be determined by active- triangulation parameters, and is given as: where N is the effective number of spatial periods that would occur over 2p radians of projection azimuth, regardless of the actual field of view of the projector.

In addition to this spatial phase, there may be an additional phase introduced by a fast sweeping modulation of the projector. This additional phase can be considered as a time-of-flight-like phase with both temporal and spatial dependence. Because this time-of-flight phase, fto _ΐ may be introduced by a projected pattern sweep and not by an overall modulation of the amplitude of the projector emission, and may be added directly to the spatial phase. The reason for this is illustrated as follows: At a particular azimuthal projection direction Q, (300) at a particular time t, assume the phase emitted by the projector changes at a constant rate, = 2nf _m , where f _m is the characteristic frequency of the projected pattern sweep, i.e., the inverse of the time that it takes to sweep the projected pattern by one full spatial period. Therefore, the phase leaving the projector in direction Q at time t is 0 _em ; _t = f _AT + 2nf _m t.

However, this may not be the phase observed at the camera at the same time t, since it may include an additional offset due to a time delay that may be introduced by the propagation of light over a finite distance from the projector to the object and back to the camera. This distance may include the distance D _p (element 125 in Fig. 1) and distance D _c (element 110 in Fig. 1). D _p and D _c are given by, for example and

D _c = Z[ tan ² /? + l] ¹ / ² .

This gives a total time of flight phase shift which can be expressed as: The total phase at the camera at time t can be expressed as:

To determine the noise performance of the method, the equation for above may be differentiated with respect to z to achieve:

By propagation of random error in the measured phase, the depth error may be given by, for example,

This equation may give the random error in z for a given random error in phase, ^s Phase error is inherent in many phase detection systems, and may arise from the level of signal (shot noise) as well as from other system-dependent sources. For equivalent phase error the proposed technique has lower depth error than the equivalent time-of-flight or active-triangulation systems, for values of Z depth. This technique therefore provides a better and lower-noise estimate of depth.

Unlike many proposed “hybrid” systems, which may perform measurements independently and then in some way combine results, the method described here is instead a single measurement. The operational flow of such a system is demonstrated in Fig. 4. At step 400, a projector (e.g., projector 115), a time-of-flight sensor (e.g., time-of-flight sensor 100) or other device may project sweeping phase data. At step 405, a sensor (e.g., time-of-flight sensor 100) may sample reflected light. In some embodiments, the samples may be offset by a baseline distance. At step 410, a computing device may calculate a phase for a time-of-flight pixel (e.g., each time-of-flight pixel available). At step 415, a computing device may remove an ambiguity from phase data (e.g., phase data determined at step 410), such as by performing a process of unwrapping, discussed further below. For example, an ambiguity may be considered an element of phase data that is statistically different from other phase data elements beyond a threshold. At step 420, a computing device may convert phase data to depth values, such as by using calibration parameters. At step 425, a computing device may output a depth map, which may be based on (e.g., may comprise) some or all of the depth values calculated at step 420.

With this system, one phase measurement from the time-of-flight sensor may produce a depth estimate; additional measurements may be used in some embodiments. This technique offers superior noise performance, which may result from having a single combined phase value from both the active- triangulation and time-of-flight features of the system. As an example of this performance, Fig. 5 illustrates a comparison of the noise performance of both time-of-flight, active-triangulation, and combined measurement techniques for the same phase noise. The plots in Fig. 5 are taken for an exemplary representative case using the above equations. The performance comparison for each system assumes equivalent system parameters held constant to a degree, for example the total light emitted may be the same. As Fig. 5 shows, the active-triangulation method has a better noise performance than the time-of-flight method at near distances. Conversely, at far distances, the time-of-flight method performs better than active-triangulation. At some distance there is a crossover where the performance curves cross each other. The proposed combined phase measurement, however, performs better than either curve at various distances.

In examples of the presently disclosed subject matter, multiple periods of a periodic pattern are projected. When multiple periods of a periodic pattern are projected, it may be necessary to remove resultant discontinuities or phase-wrap points. This process may be termed “unwrapping”. Several techniques exist to achieve the removal of phase-wrap points. In some embodiments, an active- triangulation technique may involve projecting and capturing patterns containing different spatial frequencies. For example, a pattern with 10 cycles across the field of view may be projected and captured, followed by a pattern with only two cycles across the field of view. Similarly, multiple different temporal modulation frequencies may be used with time-of-flight techniques. For example, a time-of-flight system may project and demodulate data at a modulation frequency of 100 Mhz, and may then change to a second capture taken at 10 Mhz. In both examples, a coarse depth estimate may be extracted from the lower frequency sample, which may generate a coarse but unambiguous estimate of depth, and the higher spatial frequency may then be used to increase the precision of the unambiguous depth estimate. In the proposed system, either technique may be implemented, or both techniques may be combined in any manner with any number of captures such that different sweep rates and different numbers of periods are sequentially projected and captured.

Other unwrapping techniques may be used, including those that do not include multiple captures by a single system, but instead rely on additional data inputs to correctly disambiguate depth measurements. These additional inputs may be provided by other systems of the same nature operating within the same field of view, or by other systems of different nature, such as passive-stereo depth sensors.

Unwrapping may also be implemented with no additional input, for example using computational techniques. For example, an algorithm structured to establish temporal or spatial continuity may be used (e.g., an algorithm written to execute any or all of the equations described herein).

In the fast-sweep embodiment, a projector may generate periodic patterns that can be swept at time-of-flight rates, which may be above IMhz. Several possible projector embodiments are described, which may produce such an output. These embodiments are also compatible with a slow-sweep method, described further below. Fig 6. captures one such embodiment.

In this embodiment, an array (600) of controllable emitters, which may be a vertical-cavity surface emitting laser (VCSEL) array, which may consist of multiple emitters (630), which may be evenly or unevenly spaced, and which may be addressed individually or in groups. For example, the emitter array may be driven to produce an intensity that is periodic across the array columns. For example, columns, rows, or other groupings of emitters may emit light according to a waveform (625). By adjusting the array input rapidly in time, this spatially periodic pattern may appear to sweep, such that it is also temporally periodic. In some embodiments, an optical system (605) may alter the output from these emitters. In some embodiments, the emitters may emit light across a field of view (610). For example, the optical system may introduce a degree of smoothing and/or defocus along a horizontal axis (615), and/or a spreading and/or expanding function in a vertical axis (620) such that a smooth sinusoid or other periodic intensity may exist in the horizontal dimension and may be expanded in the vertical dimension. The optical system may consist of one or more lenses, one or more diffractive optical elements, and/or other optical components. Accordingly, this embodiment may produce a sweep that is approximately continuous.

Another projector embodiment for a fast-sweep method is shown in Figs. 7 and 8. In this embodiment a similar array of addressable emitters may be used (700, 810). For example, an array (700, 810) may be angled to a position (705), which may a position within a field of possible anglings (710). The array (700) may emit light across a field of view (715), which may have a horizonal axis (715) and a vertical axis (720). Also shown is a reference center line (725).

This array (700, 810) in some embodiments may not produce this spatially periodic pattern, but may instead, output light such that it passes through an intermediate pattern-forming element (815) that modulates the light to produce the spatial structure. This intermediate element may comprise, for example, of a high contrast grating diffractive optical element (DOE) and/or an amplitude mask. For example, a DOE may comprise a transmissive plate that has been etched with an intent of modulating a phase of transmitted light. By modulating the phase of transmitted light, it may be possible to achieve zones of constructive and destructive interference which produce higher or lower intensity in the far-field, according to a desired intensity distribution pattern. For example, an amplitude mask may comprise a transmissive glass plate selectively coated with an absorptive or reflective material such as chrome. In some embodiments, a placement of the chrome may create areas where light is blocked or attenuated, and a pattern may thus be defined.

This pattern-forming element may output a spatially periodic pattern with a phase that may vary as a function of an input light position or angle. Fig. 8 illustrates an example of a system with this property. In some embodiments, an amplitude mask (815) has square-wave transmissivity such that a horizontal line of illumination passed through the mask produces a square wave output (820). The square- wave pattern of the mask may be tilted relative to the illumination array (810) such that a line of illumination that sweeps from a first portion (800) to a second portion (805) will output a square wave intensity that sweeps in phase from (820) to (825). In some embodiments an optical system may also be used, such that the sweeping of the illumination may have a reduced impact on the displacement of the projected image. In some embodiments, an emitter array with a plurality of rows may be controlled such that rows (e.g., each row) are illuminated sequentially. This may produce a spatially periodic pattern that is swept in time in incremental steps, where the number of steps may be equal to a number of rows in the emitter array, though any number of steps may be used. In other embodiments, such as to produce a continuously sweeping pattern, an array may drive multiple rows of emitters simultaneously with a pulse intensity, such as is depicted as waveform (800). This pulse may have a spatial width that extends across multiple rows of emitters. Thus, in some embodiments, by modulating adjacent rows of emitters in pulse intensity, a peak intensity of the pulse may be smoothly and continuously swept along the array, which may generate a smooth (or substantially smooth) and/or continuously (or substantially continuously) sweeping illumination output.

The ‘fast-sweep’ embodiment previously described may include a projector that sweeps a pattern at time-of-flight rates (such as above IMhz). In so doing, that embodiment outputs a light field in which the phase of the pattern may vary as a function of both angle from projector (140) and time. In some embodiments, a projector may project at least one output pattern. In some embodiments, the at least one output pattern may comprise a first and a second output pattern, and the first output pattern may vary from the second output pattern in space and time. In some embodiments, the at least one output pattern may have a frequency greater than 1MHz. In some embodiments, it may be desirable to output a projection with the same basic behavior (phase that varies with both angle and time), but may not use such a high rate of sweeping. One such embodiment, referred to as the ‘slow-sweep’, will now be described. As with the previous embodiment, this method may produce a single measurement of depth by using an algorithm that may rely on both active-triangulation and time-of-flight information.

This ‘slow-sweep’ embodiment may utilize a pattern projector that sweeps an illumination of a known pattern at predetermined rate, and may have an overall signal amplitude modulation at a higher, time-of-flight rate. For example, the pattern may sweep in space at 30 KHz, while the signal amplitude may be modulated at 200 Mhz. The spatial intensity distribution of the projected pattern may be either sinusoidal, a square wave, a trapezoidal function, a Hamiltonian function, or another periodic structure. The time-of-flight modulation may be either sinusoidal or a square wave. To demonstrate the principle, which may apply to subsequent possible embodiments, a sinusoidal function is chosen for both the time- of-flight and the active-triangulation pattern.

In this slow-sweep method, a high-speed time-of-flight signal may be modulated within a spatially structured amplitude envelope. In some embodiments, this amplitude envelope may be defined to have a specific phase relationship to the time-of-flight signal, and this relationship may involve controlled shifts during the sample measurement. With the appropriate signal, the phase measured by the time-of-flight sensor alone may be used to produce a depth estimate.

An example of this system architecture is demonstrated in Fig.l. A projector (115) may output a sweeping pattern (120). In this example, the pattern may be assumed to have a sinusoidal spatial intensity distribution, as well as an overall sinusoidal temporal time-of-flight signal amplitude modulation. The projector output ray (125) may impinge on an object in the field of view (130) and a reflected ray (110) may arrive at the time-of-flight sensor (100) that is offset by a baseline distance (135). The intensity of light arriving at the sensor may be expressed as: where m is a number, not necessarily an integer, that represents the relative phase difference between the projector time-of-flight modulation and the sensor time-of-flight demodulation. In this equation, S _m is the total light intensity incident at the sensor (taking into account reflection losses and other attenuation), at time t for a given value of m. R _s is the amplitude of the spatial modulation pattern as observed by the sensor. is the spatially and temporally constant background component of light in the projector pattern. is the phase of the spatial pattern (a function of projected direction and depth z). is the phase from time-of-flight delay (a function of projected direction and depth z). f _m is the frequency of the overall fast modulation of the projector signal. f _s is the frequency of the active-triangulation pattern sweep.

This projection is visualized in Fig. 9. It is assumed in this exemplary case that the active- triangulation pattern and the time-of-flight amplitude modulation are both sinusoidal. At one instant in time, a radial cross section from the projector may have an exemplary sinusoidal intensity waveform (900) corresponding to the active-triangulation pattern. The phase of this signal is illustrated in an exemplary waveform 905. In some embodiments, an overall time-of-flight modulation may take place at a much greater rate than the active-triangulation modulation. This is demonstrated by an exemplary waveform (910), which shows a sample taken at one position in space over time. There may be an overall signal envelope corresponding to the sweep of the active-triangulation, and, in some embodiments, multiple periods of time-of-flight modulation may exist within a single period of this envelope. These combined waveforms have a single phase value shown in an exemplary waveform (915). Because of this property, a measurement taken at a constant angle from the projector, at a single moment in time, may have an approximately periodic intensity with respect to the distance Z, seen in an exemplary waveform (920). Similar to waveform 915, this intensity may have a single value of phase, such as shown in an exemplary waveform (925). It should be appreciated that according to embodiments of the present disclosure, a phase measurement from the time-of-flight sensor may be sufficient data input for generating a depth estimate. It would be further appreciated that a resulting phase value may be influenced by properties of active-triangulation systems, such as a baseline length, as well as properties of time-of-flight systems, such as a modulation rate. With such systems, it is possible to incorporate additional measured data, for example, to improve the quality of the depth estimate. It would be still further appreciated that in existing so-called ‘hybrid’ systems, distinct measurements are used, unlike with the present embodiment, for active-triangulation and time-of-flight to yield a depth estimate.

For clarity and as an example, the previous equation is expanded and the trigonometric identity is applied to yield an equivalent expression, f j ^R e _{mn mn} +R _B [(cos (0 _roF )cos (2nf _m t + — ) - sin (f _T0R ) sin (2Tr/ _m t + — )] _Ϊ j +R _s [( _. cos ty _AT )cos(2nf _s t) - smty _si ) sm(2nf _s t)] I

^in ^~ _n \ R [

I + -y [cos(0 _roF - f _AT + 2nt(f _m - f _s ) + mn/2)] | The last two rows in this expression are a ‘phase difference’ and ‘phase sum’ cosine terms respectively, in which the time-of-flight and the active-triangulation phases are combined. Assuming the sign conventions on the phase definitions are such that the phase sum term is the desired quantity of measure, the goal in designing the projector/camera system should be to manipulate the signal and/or the measurement of the signal such that many or all terms (except for the ‘phase sum’ term in the last row and, by necessity, the constant term in the first row) average to zero over one period of the slow sweep.

As an example implementation of this technique, the previous equation is rewritten in terms of ¼ period intervals, such that time t= 0 at the start of each ¼ period cycle and the introduced integer n represents an index (0-3) representing a particular ¼ period interval. ¼ period intervals are used in this example, though it is possible to implement the same principles with other even fractional intervals, e.g. l/8th periods. The case of ¼ period intervals leads to the following expression:

From this expression, it is seen that if the term (m+n) is a constant, the last row of the above equation, or ‘sum-phase’ term, may be the same across all four quadrants of a slow scan, and this part of the signal will accumulate in the integration of the time-of-flight sensor. This criterion can be met if the value of m is quickly switched at the start of each ¼ period scan so that m+n is constant. In some embodiments, this may be accomplished by shifting the projector phase or the sensor phase at a particular rate (e.g., rapidly). While the sum-phase term may remain the same across all period quadrants, m may be chosen to force the other time-varying terms in the expression to average zero over the entire period scan, while still satisfying (m+n) remaining constant. As an example, if m=4-n, the sum-phase term is constant and other time varying terms average to zero. Assuming a time-of-flight modulation with a large number of periods in one ¼ period of the slow sweep, and a time-of-flight sensor synchronized to the overall time-of-flight projector signal modulation in a typical fashion, the total recorded signal in one time-of-flight sensor channel with m=4-n may consist only of the sum-phase term and the constant background term. So, by selecting the values of m and n to meet the above criterion, the system may record the signal components to make a direct measure of · Thus, the described method provides a measurement of the phase of each signal which may then be converted to a depth value. This measurement of the phase of each signal distinguishes this method from other techniques that capture the phase of only one signal and the amplitude of another. Because the phase of each signal may not vary with object albedo or ambient lighting conditions, this technique is robust, in particular compared to other methods that extract depth from a signal amplitude measurement.

In some embodiments, the actual time- integrated signal in each time-of-flight sensor channel may be determined, but it should be noted that the frequency of the sum-phase term is not f f _s . This has a subtle effect on the recorded time-of-flight signal. Specifically, if the time-of-flight sensor and projector are synchronized in each ¼ period scan (at the beginning, for example), the modulation and demodulation may drift an additional ¼ period out of phase by the end of each quarter period scan. In some embodiments, it may be assumed that the sensor is of a type that records phase values by selectively switching between collectors or ‘channels’ with square-save timing. Further assuming the phase relationships given in the equation for S _{n m} above, it can be seen then that the recorded signal time-of- flight sensor channels with 0° and 180° phase may be expressed as: where N _p is the number of fast time-of-flight periods integrated. For completeness, the 90° and 270° phase measurements will have signals that may be expressed as:

The total phase FT _O T _A I = <PT _O F + <P _S L may then be expressed as:

Of course, in some embodiments, such as during the signal collection, if the slightly slower demodulation is offset by a positive ¼ period with respect to the modulation at the beginning of each ¼ slow period, then the drift may be balanced around the target phase and the collected signals may be expressed as: and the total phase may be expressed as:

With either approach, the noise performance of this slow-sweep combined phase technique may be less desirable than that of the fast-sweep approach previously described (assuming equal projector intensities and collection time) because the measured signal relative to background may be less by a factor of two (or approximately two) and the overall signal level may be lower by a factor of two (or approximately two). However, in some embodiments, such as in order to make a direct comparison of these techniques, the total illumination power may be held constant. With this example of sinusoidal modulation, the output signal has a 50% duty cycle. And, in some embodiments the peak intensity may be doubled to maintain the same total illumination power. Even with a decrease in noise performance, this embodiment may still be preferable because it can be implemented with a simpler projector architecture that does not involve sweeping the pattern structure at Mhz rates.

The slow-sweep method may use any of the previously described projection systems, but because of the reduced constraints on the projector sweep rate, additional projection methods may be implemented as seen in Figs. 10, 11 and 12. As demonstrated in Fig. 10, the projector may include an illumination source such as a laser diode (1000) a variable reflective element (1010), such as a MEMS array, spinning prism mirror or mirror galvanometer, and/or projection optics (1020). By modulating the laser at time-of- flight rates, while varying the reflective element at a lower rate, an adequate projection may be provided to illuminate a desired field of view (1030).

Fikewise, another projector embodiment such as that demonstrated in Fig. 11 may be used. In some embodiments, a projector (1100) may emit light through a transmissive variable pattern forming element (1110) such as a liquid crystal or liquid crystal on silicon (FCOS), which may replace a reflective pattern generation element. In this example, an illumination (1120) may be modulated at or near time-of- flight rates while the sweeping pattern is formed at a lower rate.

Another projector embodiment, illustrated by Fig. 12, takes advantage of the fact that for the slow-sweep method, it is possible to measure the combined system phase without a smooth sweep of the pattern. In this case, it is possible to use fixed pattern forming elements with addressable emitters as shown in Fig. 12. In this embodiment, a group of emitters (1200) may be individually controlled, and each may illuminate a fixed pattern forming element, or a unique area of a common pattern forming element (1210). By modulating emitters (e.g., each emitter) at TOF rates, and switching between emitters at a lower rate, it may be possible to output an appropriate projection (1220).

In some embodiments, a single measurement of phase may be taken and converted to a depth value. This offers better performance than switching between or combining separate measurements from active-triangulation and time-of-flight. However, a system may switch from any of the modes of operation discussed herein into either an active-triangulation or time-of-flight mode. For example, in the case of the addressable emitter array shown in Fig. 6, modulating the array with emitters in phase and manipulating the collected phase data appropriately would turn the system into a pure time-of-flight system. Fikewise, it is possible to halt or slow down the overall demodulation rate of both the time-of- flight sensor and projector together to a much lower rate than normal operation, such that the impact of time-of-flight on phase could be reduced to the point that the system would operate effectively as an active-triangulation system.

In either case, one could use this switching behavior to capture data sequentially with the normal mode of operation. Data taken in either a time-of-flight or active-triangulation mode will typically be less valuable for depth estimation, since having greater measurement noise. However, this data may be used for automatic calibration. Calibration is typically done for depth capture systems. Calibration parameters are used to convert the captured data into depth data, and include parameters related to the camera and lens known as ‘intrinsics’ as well as parameters defined by the physical relationship between the projector and camera, termed ‘extrinsics’. One example of an intrinsic parameter is camera focal length. One example of an extrinsic parameter is the baseline distance between projector and camera.

Such parameters may be found for each device during an initial calibration step. After initial calibration, it is possible that the system parameters may change, thus invalidating the initial calibration. Thus, it may be valuable to automatically recalibrate (auto-calibrate) the system using techniques described herein.

One embodiment of automatic recalibration with the proposed hybrid system shall now be described. In this embodiment, the system may interrupt a method of depth sensing to capture a single time-of-flight measurement. As described earlier, this may be achieved by changing the projection and decoding signals. In some embodiments, the device may revert to normal mode of operation after capturing a single time-of-flight measurement.

From this time-of-flight measurement, the scene within the field of view may be reconstructed using a combination of calibration parameters. Due to the nature of time-of-flight, the yaw of the system (angle 300 in Fig. 3) may not significantly impact the depth reconstruction. From a subsequent measurement taken with normal operation, the scene may be reconstructed using initial measurement parameters. In some embodiments, these two measurements may then be aligned to a common coordinate system using rigid and non-rigid transformation techniques.

In some embodiments, such as when the time-of-flight measurement is aligned to common coordinates, the time-of-flight measurement may be considered as a ‘ground truth’ and calibration parameters used to reconstruct a normal measurement may be optimized to maximize the correlation in reconstructed depth between the two measurements. To implement this optimization, a computing device may iteratively test an algorithm with different values for the extrinsic parameter of ‘yaw’ described above. In some embodiments, the computing device may perform iterative tests until both reconstructions match as closely as possible (e.g., when subsequent iterations yield little to no change). The computing device may then save the determined optimal value for yaw (e.g., to a storage medium), which can be used for subsequent measurements. In this way, the system may recalibrate itself during operation. This recalibration process is possible due to the capability of the system to capture unique depth measurements with shared components.

The disclosed embodiments may be implemented in a system, a method, and/or a computer program product. For example, an algorithm may be represented as data stored on a computer readable storage medium and may be modified and/or executed by a processer. In some embodiments, a computing device (e.g., a processor) may instruct a device (e.g., a ground or air vehicle) to perform an action based on a result from an algorithm (e.g., an aspect determined from mapping a surface). In some embodiments, a computer program product may include a computer readable storage medium (or media) having computer readable program instructions thereon for causing a processor to carry out aspects of the present disclosure.

The computer readable storage medium can be a tangible device that can retain and store instructions for use by an instruction execution device. The computer readable storage medium may be, for example, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of the computer readable storage medium includes the following: a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanically encoded device such as punch-cards or raised structures in a groove having instructions recorded thereon, and any suitable combination of the foregoing.

Computer readable program instructions for carrying out operations of the present disclosure may be assembler instructions, instruction-set-architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state-setting data, or either source code or object code written in any combination of one or more programming languages, including an object oriented programming language such as Smalltalk, C++ or the like, and conventional procedural programming languages. The computer readable program instructions may execute entirely on a single device, partly on the single device, as a stand-alone software package, partly on the single device and partly on another device, or entirely on another device. In scenarios involving multiple devices, the devices may be communicably connected through any type of network, including a local area network (LAN) (e.g., within an aircraft) or a wide area network (WAN). In some embodiments, electronic circuitry including, for example, programmable logic circuitry, field-programmable gate arrays (FPGA), or programmable logic arrays (PLA) may execute the computer readable program instructions by utilizing state information of the computer readable program instructions to personalize the electronic circuitry, in order to perform aspects of the present disclosure.

This disclosure, including the figures, describes the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments. While a number of steps (e.g., operations, calculations, equation variables, aspects) are described, it should be noted that some steps may be executed substantially concurrently, iteratively, sequentially, repeatedly, in any order, or even not be executed at all. It will also be noted that the steps discussed can be implemented by special purpose hardware-based systems that perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.

The foregoing description has been presented for purposes of illustration. It is not exhaustive and is not limited to the precise forms or embodiments disclosed. Modifications and adaptations will be apparent to those skilled in the art from consideration of the specification and practice of the disclosed embodiments. Additionally, although aspects of the disclosed embodiments are described as being stored in memory, one skilled in the art will appreciate that these aspects can also be stored on other types of computer-readable media, such as secondary storage devices, for example, hard disks or CD ROM, or other forms of RAM or ROM, USB media, DVD, or other optical drive media.

Computer programs based on the written description and disclosed methods are within the skill of an experienced developer. The various programs or program modules can be created using any of the techniques known to one skilled in the art or can be designed in connection with existing software.

Moreover, while illustrative embodiments have been described herein, the scope of any and all embodiments having equivalent elements, modifications, omissions, combinations (e.g., of aspects across various embodiments), adaptations and/or alterations as would be appreciated by those skilled in the art based on the present disclosure. The limitations in the claims are to be interpreted broadly based on the language employed in the claims and not limited to examples described in the present specification or during the prosecution of the application. The examples are to be construed as non-exclusive.

Furthermore, the steps of the disclosed methods may be modified in any manner, including by reordering steps and/or inserting or deleting steps. It is intended, therefore, that the specification and examples be considered as illustrative only, with a true scope and spirit being indicated by the following claims and their full scope of equivalents.