METHODS AND APPARATUS FOR

SUPERPIXEL MODULATION WITH AMBIENT LIGHT SUPPRESSION

RELATED APPLICATIONS

[0001] This application claims the priority of the filing date of United States

Provisional Patent Application No. 62266653 filed December 13, 2015, the entire disclosure of which is herein incorporated by reference.

FIELD OF THE TECHNOLOGY

[0002] The present invention relates generally to ambient light suppression and superpixel modulation.

SUMMARY

[0003] In illustrative implementations of this invention, an imaging system

illuminates a scene with an active light source, and captures light with an image sensor. The sensitivity of image sensor pixels to light is modulated by two types of modulation, specifically: (a) what is sometimes called "curtain modulation" herein; and (b) what is sometimes called "superpixel modulation" herein.

[0004] In illustrative implementations, the curtain modulation: (a) tends to decrease the relative amount of ambient light that is captured by the image sensor; and (b) tends to increase the relative amount of reflected light from the active light source that is captured by the image sensor. Thus, a greater percentage of the light recorded by the image sensor comprises light that is emitted by the active light source and that reflects from the scene, and a lesser percentage of the light recorded by the image sensor comprises ambient light (than would otherwise occur in the absence of the curtain modulation). Thus, effectively, the curtain modulation tends to suppress the effect of ambient light, and thereby tends to improve the signal-to-noise ratio (SNR) of the image sensor.

[0005] In some cases, in the curtain modulation, a dot of light traverses a path through a scene during each frame of the image sensor. For example, the path (e.g., a zig-zag path) may include repeated movements along trajectories that are parallel to each other. For example, the path may include the dot of light making repeated vertical movements from bottom to top of the scene, with slight horizontal displacements between each vertical movement. [0006] In some cases, at any given time while the dot of light traverses this path: (a) only a narrow, moving region of the image sensor includes pixels that are sensitive to light; and (b) the remainder of the image sensor is not sensitive to light. The curtain modulation may cause the narrow region of the image sensor that includes pixels that are sensitive to light (the active region) to move in synchrony with the dot of light. The curtain modulation may thus cause the active region to include pixels which are being struck by light that is directly reflecting from the dot of light (which pixels tend to have a high intensity of incident light). Therefore, the average per pixel intensity of light incident in the active region may be higher than the average per pixel intensity of light incident in the inactive region. As a result of the curtain modulation (which causes the active region to track the moving dot of light that is projected by the active light source and which causes no light to be measured in the inactive region), the relative contribution of the active light source to the light measured by the image sensor may be greater - and the relative contribution of ambient light to the light measured by the image sensor may be less - than would occur in the absence of the curtain modulation. Thus, the curtain modulation may tend to suppress the effect of ambient light.

[0007] The active region of the image sensor (which includes pixels that are sensitive to light) may be longest along an axis that is perpendicular to the direction of movement of the dot of light. For example, if the dot of light is rising vertically in the scene, then the active region may be a narrow, horizontal portion of the image sensor that is rising vertically in the image sensor (simultaneously with the vertical rise of the dot of light in the scene).

[0008] In curtain modulation, the active region (which includes pixels sensitive to light) of the image sensor moves, in the sense that which pixels are included in the active region changes over time. Likewise, the inactive region (which is not sensitive to light) of the image sensor moves, in the sense that which pixels are included in the inactive region changes over time. However, in illustrative implementations, the image sensor and the pixels that comprise it are themselves stationary while the active region and inactive region move.

[0009] The moving region of the image sensor that is not sensitive to light (the

"inactive region) may be loosely thought of as being covered by a moving curtain. Hence the name "curtain modulation".

[0010] Here is an example of curtain modulation. In this example: (a) the dot of light moves in a zig-zag pattern over the entire scene in a single frame; (b) in this zig-zag pattern, the dot of light moves up in a vertical direction, then diagonally down and to the left, then up, then diagonally down and to the left, and so on; and (c) the dot of light is "on" during the vertical movements up, and "off during the diagonal movements down and to the left. In this non-limiting example: (a) during each vertical (up) movement of the dot of light, only a narrow, horizontal region of the image sensor is active (i.e., includes pixels that are then sensitive to light), and the remainder of the image sensor is not sensitive to light; (b) at any given time, the active region consists of a set of adjacent horizontal rows of pixels in the image sensor; (c) this active, horizontal region moves up the image sensor as the dot of light moves vertically up in the scene, such that at any given time, the active region includes pixels of the image sensor that are receiving light directly reflecting from scene (which tend to have a higher per pixel intensity of incident light than do other pixels); and (d) which rows of pixels are in the active region changes over time as the horizontal active region moves up the image sensor.

[0011] As noted above, in illustrative implementations, both curtain modulation and superpixel modulation control the sensitivity of pixels to light.

[0012] The superpixel modulation involves modulating the sensitivity (to light) of different pixels in each superpixel of the image sensor by separate modulation signals, as described in more detail below. The superpixel modulation causes the image sensor to capture (albeit at a lower spatial resolution), in a single frame, information that would otherwise require many frames to acquire with conventional structured illumination.

[0013] For example, the rapid data acquisition achieved by superpixel modulation may facilitate 3D scanning of a moving object. Conventional structured illumination requires many frames of data in order to determine depth, and thus suffers from distorted readings when attempting to 3D scan a moving object, because the object moves from frame to frame. In contrast, superpixel modulation facilitates the determination of depth from a single frame, and thus mitigates these distortions.

[0014] In illustrative implementations, the superpixel modulation is achieved as follows: An image sensor includes superpixels. Each superpixel comprises multiple pixels, such that each pixel in the superpixel is a horizontal, vertical or diagonal neighbor of at least one other pixel in the superpixel. The pixels of the image sensor also include pixel sets. Each pixel set comprises multiple pixels to which a single superpixel modulation signal is applied. This superpixel modulation signal contols the sensitivity of the pixels in a pixel set to light. Each of the pixel sets is modulated by a separate superpixel modulation signal. Each superpixel includes at least one pixel from each of the pixel sets. Thus, in superpixel modulation, the sensitivity (to light) of different pixels in each superpixel are separately and simultaneously modulated.

[0015] Here is a first example of superpixel modulation: In this first example of superpixel modulation: (1) an image sensor includes n pixel sets; (2) each superpixel in the image sensor has n pixels, such that each pixel in the superpixel is an element of a different pixel set; and (3) n separate modulation signals are simultaneously applied to the pixels in the image sensor, one separate modulation signal per pixel set. In this example, for each superpixel, n separate modulation signals are simultaneously applied to the n pixels of the superpixel, one modulation signal per pixel.

[0016] In this first example of superpixel modulation, the n separate modulations signals may together comprise an n-b t binary code, such as an n-b\t binary gray code. Each of the modulation signals may correspond to a bit of the code. Each of the n pixel sets may be modulated by a different modulation signal, and thus, at any give time, the state of each of the n pixel sets may correspond to a bit of the /7-bit code. Likewise - since each of the n pixels in a superpixel may be from a different pixel set - at any give time, the state of each of the n pixels in a superpixel may correspond to a bit of the n-b\t code. All 2 ⁿ permutations of the n-bit code may be applied at different times during each camera frame. Thus, each camera frame may be divided into 2 ⁿ time intervals (timeslots). Each permutation of the code may correspond to a unique timeslot during a single camera frame.

[0017] For example, at any given timeslot (out of the 2 ⁿ timeslots during a single camera frame), a unique permutation of states of the n separate modulation signals may occur. Thus, at any given timeslot (out of the 2 ⁿ timeslots during a single camera frame), a unique permutation of states of the n pixels of each superpixel may occur. The states of each pixel may be binary: either sensitive to light or not sensitive to light.

[0018] Here is another example of superpixel modulation: (1) each superpixel has n pixels; (2) n separate modulation signals are applied to the n pixels, one modulation signal per pixel; (3) together the n modulations signals comprise an 9-bit binary gray code; and (4) each of the n modulation signals correspond to a bit of the code.

[0019] Thus, in illustrative implementations, superpixel modulation comprises a set of separate modulation signals simultaneously modulating a plurality of pixels in a superpixel, such that each pixel in the superpixel is modulated by a modulation signal that causes sensitivity (to light) of the pixel to vary over time. [0020] In some implementations of this invention, superpixel modulation is used for

3D scanning, i.e., for detection of the 3D position of points in a scene.

[0021] In some cases, the 3D scanning with superpixel modulation is implemented as follows: Rotating mirrors cause a dot of laser light to move in a path across a scene during each camera frame. Because the mirrors rotate, the angle of the laser beam varies as a function of time during each camera frame. The mirrors are at particular angles when the dot of laser light hits the point in the scene from which light directly reflects to that pixel. At these particular angles of the laser beam, the incident light intensity for that pixel reaches a maximum for that frame. The superpixel modulation affects the pixel's response to incident light, and thereby uniquely identifies (i) the timeslot during which the maximum intensity of incident light strikes the pixel and (ii) the orientation of the laser beam during this timeslot. A computer uses the timeslot to access a lookup table to determine the 3D position of the scene point. The lookup table is computed for each pixel by tri angulation, during earlier calibration. Specifically, in some cases, the image sensor, rotational axis of the rotating mirror, and pixel ray for the superpixel are fixed relative to each other. The pixel ray is the half-line along which light travels in order to reach the center of the superpixel. A computer may calculate the direction of the laser beam for a given superpixel, based on the timeslot at which incident light for the given superpixel reaches a maximum. The 3D scene point that reflects light to the given superpixel is positioned at the intersection of (i) the pixel ray for the given superpixel and (ii) a vertical plane that is parallel to, and that intersects, the laser beam. A computer may calculate the 3D coordinates of this intersection (of the pixel ray and the vertical plane) by triangulation, based on the known position of the superpixel, pixel ray for the superpixel, and light source and based on the computed orientation of the laser beam.

[0022] As noted above, in illustrative implementations of this invention, both curtain modulation and superpixel modulation involve modulating the sensitivity of pixels in an image sensor to light. The sensitivity of the pixels to light may be modulated either: (a) by controlling the storage of electric charge in the image sensor; or (b) by selectively attenuating transmitted or reflected light, before the light reaches the image sensor.

[0023] In some implementations, the sensitivity of a pixel to incident light is modulated by controlling the storage of electric charge in an image sensor. Modulation of electric charge storage is achieved by using either a single storage device per pixel, or by using multiple storage devices per pixel. [0024] In single storage cases, each pixel transfers charge to only one storage device during the exposure time of a single camera frame. The storage device receives charge only from that pixel, and not from any other pixels, during the exposure time of a single camera frame. Modulation signals cause a photodiode in each pixel to transfer charge to the single storage device multiple times during a single camera frame, and to drain charge, without saving it for later readout, at multiple other times during the same camera frame.

[0025] In multiple storage cases, each pixel transfers charge to multiple storage devices during the exposure time of a single camera frame. These multiple storage devices receive charge only from that pixel, and not from any other pixels, during the exposure time of a single camera frame. Modulation signals cause a photodiode in each pixel to transfer charge to different storage devices, out of the multiple storage devices, at multiple times during the exposure time of a single camera frame, or to drain charge, without saving it for later readout, at multiple other times during the same camera frame.

[0026] In some implementations, the sensitivity of a pixel to light is modulated by selectively attenuating transmitted or reflected light, before the light reaches the image sensor. A transmissive or reflective SLM (spatial light modulator) selectively attenuates light from a scene, and thereby selectively controls the intensity of light incident on a pixel. For example, in some cases, a reflective SLM selectively reflects light from the scene, causing different pixels to receive reflected light from the scene at different times during a single camera exposure. Likewise, in other cases, a transmissive SLM selectively transmits light from the scene, causing different pixels to receive reflected light from the scene at different times during a single camera exposure. For example: (a) the reflective SLM may comprise a DMD (digital micromirror device) or a reflective LCOS (liquid crystal on silicon) device; and (b) the transmissive SLM may comprise an LCD (liquid crystal device) or ferromagnetic shutters.

[0027] These two methods of controlling sensitivity of a pixel to light may be employed for either curtain modulation or superpixel modulation. Thus, in illustrative implementations: (a) curtain modulation may be implemented by controlling storage of electric charge in the image sensor ("storage control") or by controlling attenuation of transmitted or reflected light ("attenuation control"); and (b) superpixel modulation may be achieved by storage control or attenuation control.

[0028] Which approach - storage control or attenuation control - is employed for superpixel modulation and curtain modulation, respectively, may vary, depending on the particular implementation of this invention. For example: (a) in some cases, both superpixel modulation and curtain modulation are achieved by controlling storage of electric charge in the image sensor; (b) in some cases, both superpixel modulation and curtain modulation are achieved by a single reflective SLM selectively reflecting light to the image sensor; (c) in some cases, both superpixel modulation and curtain modulation are achieved by a single transmissive SLM selectively transmitting light to the image sensor; (d) in some cases, superpixel modulation is achieved by controlling storage of electric charge in the image sensor and curtain modulation is achieved by an SLM selectively attenuating light before it reaches the image sensor (e.g., by selectively reflecting or transmitting light); (e) in some cases, curtain modulation is achieved by controlling storage of electric charge in the image sensor and superpixel modulation is achieved by an SLM selectively attenuating light before it reaches the image sensor (e.g., by selectively reflecting or transmitting light); (f) in some cases, superpixel modulation is achieved by a first reflective SLM selectively reflecting light and curtain modulation is achieved by a second reflective SLM selectively reflecting light, such that incoming light is selectively attenuated before reaching the image sensor; and (g) in some cases, superpixel modulation is achieved by a first transmissive SLM selectively transmitting light and curtain modulation is achieved by a second transmissive SLM selectively transmitting light, such that incoming light is selectively attenuated before reaching the image sensor.

[0029] In illustrative implementation of this invention, the curtain modulation suppresses the effect of ambient light and thus improves SNR and the accuracy of data that is acquired by superpixel modulation in a single camera frame. This, in turn, may yield more accurate 3D scanning of an object, even when the object is moving.

[0030] The description of the present invention in the Summary and Abstract sections hereof is just a summary. It is intended only to give a general introduction to some illustrative implementations of this invention. It does not describe all of the details of this invention. This invention may be implemented in many other ways. Likewise, the description of this invention in the Field of the Technology section is not limiting; instead it identifies, in a general, non-exclusive manner, a field of technology to which exemplary implementations of this invention generally relate.

BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 is a diagram that shows a matrix of superpixels. [0032] Figure 2 is a diagram that shows individual pixels within superpixels.

[0033] Figure 3 is a diagram that shows five different superpixel modulation signals that are simultaneously applied to five different pixel sets.

[0034] Figure 4 is a conceptual diagram, illustrating how superpixel modulation signals control the response of pixels to incident light.

[0035] Figures 5, 7 and 9 are each a circuit schematic for a portion of a CMOS pixel, which pixel has one floating diffusion storage device. Figures 6, 8 and 10 are timing diagrams for Figures 5, 7 and 9, respectively.

[0036] Figure 11 A, Figure 1 IB and Figure 11 C are conceptual diagrams of a superpixel, in which each pixel has multiple storage devices. In Figure 11 A, the same transfer gates are employed for superpixel modulation and curtain modulation. In Figures 1 IB and 11C, different transfer gates are employed for superpixel modulation and for curtain modulation.

[0037] Figure 12 a circuit schematic for a portion of a CMOS pixel that has three storage devices.

[0038] Figure 13 is a conceptual diagram of CCD pixels, in which each pixel has two storage devices.

[0039] Figure 14 is a conceptual diagram of CCD pixels, in which each pixel has four storage devices.

[0040] Figure 15 is a flowchart for modulation of storage using an n-ary code, such that all of the photodiode charge is stored for later readout.

[0041] Figure 16 is a flowchart for modulation of storage using an n-ary code, such that only a portion of the photodiode charge is stored for later readout.

[0042] Figure 17 is a flowchart for decoding an n-ary code, after storage modulation in which only a portion of the photodiode charge is stored for later readout.

[0043] Figure 18 is a flowchart for decoding an n-ary code, after storage modulation in which all of the photodiode charge is stored for later readout.

[0044] Figure 19 is diagram that illustrates three examples of analog modulation signals.

[0045] Figure 20A shows a reflective SLM that selectively reflects light from a scene and that is controlled by the logical AND of the curtain modulation and superpixel modulation. [0046] Figure 20B shows a reflective SLM that performs curtain modulation and another reflective SLM that performs superpixel modulation.

[0047] Figure 21 A shows a DMD that selectively reflects light from a scene and that is controlled by the logical AND of the curtain modulation and superpixel modulation array that selectively reflects light from a scene.

[0048] Figure 2 IB shows light from a scene traveling to a DMD array that performs superpixel modulation, then to a DMD that performs curtain modulation, and then to an image sensor.

[0049] Figure 21C shows light from a scene traveling to a DMD array that performs curtain modulation, then to a DMD that performs superpixel modulation, and then to an image sensor.

[0050] Figure 22A shows a transmissive SLM that selectively transmits light from a scene and that is controlled by the logical AND of the curtain modulation and superpixel modulation.

[0051] Figure 22B shows a transmissive SLM that performs curtain modulation and another transmissive SLM that performs superpixel modulation.

[0052] Figure 23 A shows an LCD that selectively transmits light from a scene and that is controlled by the logical AND of the curtain modulation and superpixel modulation.

[0053] Figure 23B shows light from a scene traveling to an LCD that performs curtain modulation, then to a LCD that performs superpixel modulation, and then to an image sensor.

[0054] Figures 24A, 24B and 24C are each a diagram of an optical system that includes a transmissive SLM.

[0055] In Figure 24 A, light from a scene travels through an imaging lens, then through a transmissive SLM, and then to an image sensor.

[0056] In Figure 24B, light from a scene travels through a transmissive SLM, then through an imaging lens, and then to an image sensor.

[0057] In Figure 24C, light from a scene travels through an imaging lens, then through a transmissive SLM, then through a relay lens, and then to an image sensor.

[0058] Figures 24D, 24E, 24F, 24G and 24H are each a diagram of an optical system that includes two transmissive SLMs: specifically, an SLM that performs superpixel modulation and an SLM that performs curtain modulation. [0059] In Figure 24D, light from a scene travels through an imaging lens, then through a transmissive SLM that performs curtain modulation, then through a transmissive SLM that performs superpixel modulation, and then to an image sensor.

[0060] In Figure 24E, light from a scene travels through an imaging lens, then through a transmissive SLM that performs superpixel modulation, then through a

transmissive SLM that performs curtain modulation, and then to an image sensor.

[0061] In Figure 24F, light from a scene travels through a transmissive SLM that performs curtain modulation, then through a transmissive SLM that performs superpixel modulation, then through an imaging lens, and then to an image sensor.

[0062] In Figure 24G, light from a scene travels through a transmissive SLM that performs superpixel modulation, then through a transmissive SLM that performs curtain modulation, then through an imaging lens, and then to an image sensor.

[0063] In Figure 24H, the optical system also includes a relay lens. Figure 24H shows positions in which transmissive SLM(s) may be located in the optical system.

[0064] Figure 25 is a conceptual diagram that shows an example of prior art, time- multiplexed structured illumination.

[0065] In Figures 26, 27A, 27B, and 27C, a moving beam of light illuminates a scene with a moving dot of light.

[0066] Figure 26 illustrates a moving beam of light produced by controlled movement of a light source.

[0067] Figure 27A shows a mirror that reflects a moving beam of light.

[0068] Figure 27B shows a projector that projects a moving beam of light.

[0069] Figure 27C shows an array of directional light sources that, taken together, emit a moving beam of light.

[0070] Figure 28 is a block diagram of hardware components.

[0071] Figure 29A is a diagram of a 3D scanning system.

[0072] Figure 29B is a diagram that shows four examples of light paths that may result from rotating two mirrors.

[0073] Figure 30 is a flowchart that shows steps in a method of 3D scanning, in which different pixel sets are separately modulated.

[0074] Figure 31 is a flowchart that shows steps in a method of 3D scanning, in which a timeslot is calculated. [0075] Figure 32 is a flowchart that shows steps in a method of calibrating an image system.

[0076] Figure 33 is a diagram that shows four examples of circles used in an optical calibration pattern.

[0077] Figure 34 is a diagram that shows examples of use of calibration patterns.

[0078] Figure 35 is a diagram of a Z-shaped trajectory of a dot of light in a scene, produced by a light beam moving through the scene during a single frame.

[0079] Figure 36 A, Figure 36B, and Figure 36CC show the vertical position of an active region of the image sensor at different times during a vertical rise of a light beam.

[0080] Figure 37 shows a horizontally-oriented active region 601 that is "wrapped" around the top and bottom of the image sensor.

[0081] Figures 38A, 38B, 38C, 38D, 38E, 38F, 38G, 38H, 381, and 38J show an example of curtain modulation, in which an active region 3801 of pixels moves down an image sensor, wraps around to the top of the image sensor, and moves down the image sensor again.

[0082] Figure 38K shows a set of signals for controlling the curtain modulation shown in Figures 38A-39J.

[0083] Figure 39A and Figure 39B show a rotating, transmissive SLM that performs curtain modulation and is in the general shape of a right circular cylinder. Figure 39A shows a view along the axis of the cylinder. Figure 39B shows a side view of the cylinder.

[0084] Figure 39C and Figure 39D show a moving, flexible, transmissive SLM that performs curtain modulation and that surrounds a camera. Figure 39C shows a view that is perpendicular to an optical axis of the camera. Figure 39D shows a side view of the flexible SLM.

[0085] Figure 40 shows an example of curtain and superpixel modulation signals that control a pixel's sensitivity to light.

[0086] Figure 41 is a flowchart that shows a method for curtain modulation and superpixel modulation.

[0087] Figure 42 is another flowchart that shows a method for curtain modulation and superpixel modulation.

[0088] Figure 43 is a flowchart that shows a method for controlling curtain modulation. [0089] The above Figures (except Figure 25) show some illustrative implementations of this invention, or provide information that relates to those implementations. However, this invention may be implemented in many other ways.

DETAILED DESCRIPTION

[0090] In illustrative implementations of this invention, both curtain modulation and superpixel modulation are performed. The curtain modulation suppresses ambient light and thus improves SNR and the accuracy of data that is acquired by the camera. The superpixel modulation allows data to be acquired (albeit at a lower spatial resolution) in a single camera frame, that would otherwise require multiple camera frames using conventional structured illumination. This, in turn, may yield more accurate 3D scanning of an object, even when the object is moving. Thus, the curtain modulation improves the accuracy of data acquired by superpixel modulation.

Curtain Modulation

[0091] In illustrative implementations, in curtain modulation, the light source emits a beam of light (e.g., a beam of laser light or a beam of collimated LED light). The beam of light is scanned through the entire scene in a rapid motion during each single frame. In some implementations, the scan pattern is a "Z"-pattern. For example, in some cases, the scan pattern during a single frame is as follows: The light beam starts at the bottom of the scene at the beginning of the frame, rises vertically to the top of the scene, then is turned off while the scan moves diagonally down and to the left, then is turned on again. This movement (i.e., vertical rise, followed by diagonal shift down and to the left) is repeated numerous times, until the left-top corner of the scene is reached at the end of the frame. In this movement, each vertical rise (after the initial rise) is horizontally displaced to the left from the preceding vertical rise. Thus, in the above example, the Z-shaped scan pattern moves from right to left, with the light beam "on" during vertical rise and "off during diagonal movement that involves horizontal displacement. Other scan patterns may be used, including scans in which: (a) the light beam is "on" during vertical motion and the vertical motion is downward; (b) the light beam is "on" during vertical motion and the Z-shaped pattern moves from left to right; or (c) the Z pattern is rotated by 90 degrees clockwise or counterclockwise, such that the light beam is "on" during horizontal motion and is off during diagonal movement that includes vertical displacement. [0092] In other implementations, the beam of laser light is moved in a different pattern. For example, the dot may move in a bottom-up, bottom-up line motion multiple times, and only then it is turned off and shifts horizontally to another column. In some cases, the beam of laser light rises to the top of the scene, then moves a very short distance horizontally, then descends to the bottom of the scene, then moves a very short distance horizontally, and repeats this pattern (up-sideways-down-sideways) until the beam traverses the entire scene.

[0093] In yet other implementations, the pattern of movement is rotated by 90 degrees, so that rows become columns and vice versa. The same holds for orientation of the active region of the image sensor: The activation pattern is rotated so that rows become columns and vice versa.

[0094] In illustrative implementations, in curtain modulation, the scan of the light beam is synchronized with the changes in sensitivity of the pixels of the image sensor. As the light beam sweeps along a line of the scene, the active region of pixels (which includes pixels that are sensitive to light) - sweeps across the image sensor. At any given time, the active region of pixels comprises a set of one or more lines of pixels in the image sensor. In many cases, each of these lines extend over the entire width (if the lines are rows) or entire height (if the lines are columns) of the image sensor. At any given time when the light beam is moving, the one or more lines of pixels in the active region are oriented perpendicular to the direction of motion of the light beam. As the light beam moves, the active region of the image sensor moves in the same direction as the light beam.

[0095] To illustrate: In a first example, the light beam is "on" during vertical rise, and the lines of pixels in the active region of the image sensor are rows that are oriented horizontally. As the light beam rises, this horizontally-oriented active region of the image sensor also rises. At any given time during the rise of the light beam, the height of the dot of light illuminated by the light beam corresponds to the vertical position (in pixel coordinates) of this horizontally-oriented active region of the image sensor.

[0096] In a second example, the light beam is on during horizontal movement to the right, and the lines of pixels in the active region of the image sensor are columns that are oriented vertically. As the light beam moves horizontally to the right, this vertically-oriented active region of the image sensor also moves to the right. At any given time during the horizontal movement of the light beam, the horizontal position of the dot of light illuminated by the light beam corresponds to the horizontal position (in pixel coordinates) of this vertically-oriented active region of the image sensor.

[0097] In illustrative implementations, in curtain modulation: (a) the image sensor does not move; and (b) the rise (or other movement) of the active region of the sensor occurs as different pixels in the sensor become sensitive.

[0098] In curtain modulation, the sweep of the active region of the image sensor is synchronized with the sweep of the light beam across the scene during a single frame. The active region of the image sensor tracks the movement of the light beam, such that at any given time, the intensity of reflected light from the light beam is greater in at least part of the active region of the image sensor than in any other region of the image sensor. The effect is that, over the course of the scan during a single frame, the sensor captures a greater proportion of reflected light from the light beam and a lesser proportion of ambient light, than the sensor would capture if the entire sensor were sensitive to light during the entire scan.

[0099] Figure 35 is a diagram of a sawtooth-shaped trajectory of a dot of light during curtain modulation, in an illustrative implementation of this invention. This sawtooth trajectory is produced by a light beam sweeping through the scene during a single frame. Each "Z" in the sawtooth trajectory includes a vertical movement up and a diagonal movement down and to the left. For example, one of the vertical movements passes through points 3501, 3502 and 3503 of the scene.

[00100] Figure 36 A, Figure 36B, and Figure 36CC show the vertical position of a horizontally-oriented active region of the image sensor at different times during a vertical rise of a light beam, in an illustrative implementation of this invention. This vertical rise occurs during curtain modulation. As the dot of light in the scene rises, the horizontally- oriented active region 3601 of the image sensor rises with it. When the dot of light is at point 3501 in the scene (as shown in Figure 35), the active region 3601 is in the position shown in Figure 36A. When the dot of light is at point 3502 in the scene (as shown in Figure 35), the active region 3601 is in the position shown in Figure 36B. When the dot of light is at point 3503 in the scene (as shown in Figure 35), the active region 3601 is in the position shown in Figure 36C. At any given time during curtain modulation, a portion of the image sensor is active, and the remainder of the image sensor is inactive. Thus, at any given time during curtain modulation, the image sensor comprises an active region 3601 and an inactive region 3602. [00101] In some cases, in curtain modulation, the active region of the image sensor, loosely speaking, "wraps around" the edges of the image sensor. For example, the active region may rise vertically, such that as the active region rises above the upper edge of the image sensor, the active region reappears at the bottom of the image sensor and rises from the bottom of the image sensor.

[00102] Figure 37 shows an example of an active region of pixels in curtain modulation, in an illustrative implementation of this invention. Figure 37 shows a time at which the active region 3601 has risen so far that the active region is, loosely speaking, "wrapped around" the bottom and top edges of the image sensor. Specifically, at the time shown in Figure 37: (a) the active region 3601 of the image sensor comprises one or more horizontal rows of pixels that are at the top of the image sensor and one or more horizontal rows of pixels that are at the bottom of the image sensor; and (b) the inactive region 3602 comprises rows of pixels that are vertically between the upper and lower portions of the active region 3601.

[00103] Figures 38A - 38J show an example of curtain modulation, in which an active region 3601 of pixels moves down an image sensor, wraps around to the top of the image sensor, and moves down the image sensor again, in an illustrative implementation of this invention. Figures 38A-38J are in temporal order, with Figure 38A occurring before Figure 38B, and so on. The active region 3601 is in a different vertical position in each of these Figures 38A-38J.

[00104] Figure 38K shows a set of signals 3802 for controlling the curtain modulation shown in Figures 38A-38J, in an illustrative implementation of this invention. For ease of illustration, in the example shown in Figures 38A-38K, the image sensor comprises ten rows and ten columns of pixels (in actual practice, however, the number of rows and columns of pixels may be much larger than ten). Figure 38K shows ten time steps 3801 of the signals. At any given time, a pixel in an image sensor is sensitive to light when both the curtain modulation signal and the superpixel modulation signal for that pixel is high. For example, in Figure 38K, a pixel in row 1 (i.e., the top row) would be sensitive to light when both (a) the superpixel modulation signal for that pixel is high and (b) simultaneously the curtain modulation signal for row 1 is high. In the example shown if Figure 38K, the curtain modulation signal is high during time periods 5, 6, 7.

[00105] In some cases, a rotating, transmissive SLM is employed for curtain modulation. For example, the rotating, transmissive SLM may comprise: (a) a rotating cylinder that surrounds a light source or camera of the imaging system; or (b) a flexible transmissive SLM that forms a physical, rotating loop which surrounds a light source or camera of the imaging system.

[00106] Figure 39A and Figure 39B shown an SLM 3901 that performs curtain modulation, in an illustrative implementation of this invention. This SLM 3901 is in the general shape of a right circular cylinder. This cylindrical SLM 3901 includes opaque regions (e.g., 3911, 3913) and transparent regions (e.g., 3912, 3914), and thus selectively transmits light. This cylindrical SLM 3901 rotates. A light source 3915 (e.g., a laser) is positioned inside the cylinder. At the time shown in Figure 39 A: (a) light ray 3931 from the light source is blocked by an opaque region 3913 of the cylinder; and (b) light ray 3930 from the light source passes through a transparent region 3914 of the cylinder and travels to the scene being imaged. Figure 39A shows a view along the axis of the cylinder. Figure 39B shows a side view of the cylindrical SLM 3901 and of an actuator 3940 (e.g., electric motor). The actuator actuates the rotational motion of the cylindrical SLM 3901.

[00107] Figure 39C and Figure 39D show a moving, flexible, transmissive SLM that performs curtain modulation, in an illustrative implementation of this invention. The flexible, transmissive SLM 3951 may comprise, for example, a bendable sheet of material. This flexible SLM 3951 includes opaque regions (e.g., 3961, 3963) and transparent regions (e.g., 3962, 3964), and thus selectively transmits light. This flexible SLM 3951 moves in a loop around three rollers 3971, 3972, 3973. Actuators (e.g., electric motors) 3991, 3992 and 3993 rotate rollers 3971, 3972 and 3973, respectively, and thereby actuate motion of the transmissive SLM 3951. A camera 3950 is positioned inside the flexible SLM 3951. At the time shown in Figure 39C: (a) light ray 3981 from the scene is blocked by an opaque region 3961 of the flexible SLM 3951, before reaching the image sensor; and (b) light ray 3980 from the scene passes through a transparent region 3964 of the flexible SLM 3951 and travels to the image sensor. Figure 39C shows a view of the flexible SLM that is perpendicular to an optical axis of the camera 3950. Figure 39D shows a side view of the flexible SLM.

[00108] In some cases, the curtain modulation (for ambient light suppression) is controlled as follows: According to user settings and parameters (such as scene type and noise level, and calibration data), the parameters of light projection and of the moving active region in image sensor are calculated. These include the width of the active region and speed of movement of the active region. At each time step, the position of the laser dot is evaluated. If necessary, the laser dot is moved by executing commands to the projection unit, such as changing the angle of the mirror(s). This makes the dot move. Also, at each time step, if necessary, commands are sent to the image sensor to move the active region. Thus, the active region is repositioned to capture light reflected from the scene illuminated by the laser dot.

[00109] Figure 43 is a flowchart of a method for controlling curtain modulation, in an illustrative implementation of this invention. In Figure 43, the active region of the image sensor is called the "rolling curtain window" or "curtain window". In the example shown in Figure 43, the method for controlling curtain modulation includes the following steps:

Calculate parameters of light projection and curtain window, including width of rolling curtain window and speed of the curtain window's movement. When doing so, take into account calibration data and user settings and parameters (Step 4301). At each time step, the position of the laser dot is evaluated and the positions of the laser dot and curtain window are synchronized (Step 4302). The angle at which the laser beam is projected is controlled, such as by controlling the angle of mirrors (Step 4303). The movement of the rolling curtain window is controlled, such as by triggering a window shift (Step 4304).

[00110] In illustrative implementations, the moving dot of illumination may be produced in many different ways. For example, any of the following types of light sources may produce a moving dot of light for curtain modulation: (a) mirror galvanometer; (b) micromirror or an array of micromirrors; (c) oscillating mirror (e.g., which is kept at harmonic oscillations); (d) projector; and (e) multiple sources of light that are arranged at different angles and turned off and on.

[00111] One of these types of light sources may be used for actuating the light beam in the x direction, and the same or different type of light source may be used for actuating the light in the .y direction. For example, in one embodiment, an oscillating mirror is used to provide movement along the_y-axis, and a galvo mirror is used to actuate in the x-axis.

[00112] In many cases, one mirror is arranged to have its axis of rotation perpendicular to the other mirror's axis of rotation.

[00113] This invention is not limited to laser light. Alternatively, a different source of light may be used. For example, collimated light may be used. Collimated LED light, which is not coherent, may be advantageous because incoherent light does not create speckle patterns. Speckle patterns, in the context of intensity measurements, are often considered problematic because the irregular variation in intensity of light influences the value readings. [00114] In curtain modulation, an input/output device may accept input from a human that selects the direction of the movement of the scan of the light beam and of the active region of the sensor. Alternatively, an external computer may generate a control signal that selects this direction of movement.

[00115] In curtain modulation, a dot of light may traverse the scanning pattern for the entire scene in a single camera frame. Alternatively, in some cases, the dot of light may traverse only a portion of the scene (e.g., a sweep along a single column or row) during a single camera frame.

[00116] In some cases, in curtain modulation, when the edge of the active region touches a given pixel so that given pixel leaves the active region, the charge accumulating in photodiode is stopped from transfer to storage(s) and the pixel becomes inactive. In other cases, a pixel is flushed all the time when it is not in active region (not only when it enters the active region).

[00117] In some cases, the movement of the start-row and end-row indexes occurs in discrete steps, where each discrete step moves multiple rows. The size of step may be defined as a constant, or may be provided externally. Alternatively, in some cases, the light beam moves in a horizontal direction when the light beam is on, and the index pointers are to a start-column-index and stop-column index.

[00118] In illustrative implementations, the orientation of the camera relative to the light source is calibrated. In some cases, epipolar lines are parallel to the image sensor bottom and top edge, and thus parallel with the image sensor rows. However, if an imperfect alignment and/or synchronization occur, the epipolar line may not be parallel, and may cross several rows on the sensor.

[00119] In some implementations, in curtain modulation, the active region is a single row (or column) of pixels or superpixels. Alternatively, in curtain modulation, the size of the active region may be two or three rows (or columns) of pixels or superpixels so as to accommodate possible misalignment and variations in synchronization. In some cases, the size of the active region is more than three rows (or columns) of pixels or superpixels. For example, this may be desirable to permit sufficient time to accomplish transfer of charge from a photodiode to storage (or to flush). The size of the active region is configurable within limits.

[00120] In some implementations, in curtain modulation, a triggering synchronization signal (that controls when the active region moves) is provided externally. This enables a user to control the speed of scan. In some cases, a triggering synchronization signal (that controls when the active region moves) is generated within the image sensor. This simplifies the communication interface with the image sensor.

[00121] In some cases, at any time that the dot of light is projected on the scene, each pixel in a region of the imaging sensor (the "direct reflection region") is receiving light that is reflecting directly from the scene. The intensity of incident light tends to be higher in the direct reflection region than in most other pixels. This direct reflection region may repeatedly change position relative to the imaging sensor, due to the dot of light changing position in the scene. The active region of the imaging sensor may also repeatedly change position relative to the imaging sensor. The changes in position of the direct reflection region and active relative to the imaging sensor may occur by changing which pixels are members of the direct reflection region and window region, respectively. At any time that the dot of light is projected on the scene, the active region may include substantially all of the pixels in the direct reflection region and may also include other pixels. The number of pixels in the active region, at any given time, may be much less than the total number of pixels in the imaging sensor. The curtain modulation may be high for all pixels in the active region and low for all other pixels in the imaging sensor.

[00122] In some cases, the active region comprises: (a) a set of rows of pixels of the image sensor, such that each row in the set of rows is adjacent to another row in the set of rows; or (b) a set of columns of the image sensor, such that each column in the set of columns is adjacent to another column in the set of columns. In some cases, the active region sweeps repeatedly over the imaging sensor (e.g., repeatedly during each camera frame), such that (i) at least a portion of each row of pixels of the imaging sensor is included in the active region during a portion of each sweep, or (ii) at least a portion of each column of pixels of the imaging sensor is included in the active region during a portion of each sweep.

Superpixel Modulation: Superpixels and Pixel Sets

[00123] In illustrative implementations of this invention, superpixel modulation involves simultaneously applying separate modulation signals to different pixels in each superpixel of an image sensor.

[00124] In illustrative implementations, an image sensor includes superpixels and pixel sets. Each superpixel includes at least one pixel from each of the pixel sets. In superpixel modulation, a separate modulation signal is applied to each of the different pixel sets. [00125] In many cases: (a) each pixel in a superpixel is from a different pixel set, and (b) thus, in superpixel modulation, a separate modulation signal is applied to each pixel in a superpixel. For example, if there are n pixel sets and each superpixel consists of n pixels (each from a different pixel set), then, for each given superpixel, the superpixel modulation may comprise simultaneously applying n separate modulation signals to the given superpixel, each to a different pixel in the given superpixel.

[00126] Figure 1 is a diagram that shows a matrix of superpixels, in an illustrative implementation of this invention. In the example shown in Figure 1, an image sensor comprises matrix 100 of superpixels, such as superpixels 101, 102, 103, 104. The matrix 100 has r rows and c columns. The total number of superpixels in the matrix is r*c=k.

[00127] Figure 2 is a diagram that shows individual pixels within superpixels, in an illustrative implementation of this invention. Figure 2 shows four superpixels 251, 252, 253, 254 in the upper left corner of a matrix of superpixels. Each of the four superpixels comprises 9 pixels. For example, superpixel 251 comprises pixels 211, 212, 213, 214, 215, 216, 217, 218, 219.

[00128] In the example shown in Figure 2, there are nine pixel sets. The pixels in the first pixel set are labeled " 1 ". Likewise, the pixels in the second, third, fourth, fifth, sixth, seventh, eighth and ninth pixel sets are labeled "2", "3", "4"', "5", "6", "7", "8", and "9", respectively. (These numerical labels are used in Figure 2 to facilitate explanation; but in actual practice, the pixels are not labeled in this manner.)

[00129] In Figure 2, each superpixel includes one pixel from each set.

[00130] In Figure 2, each pixel in a particular superpixel is a horizontal, vertical or diagonal neighbor of at least one other pixel in the particular superpixel. For example, superpixel 251 includes nine pixels, including, among others, pixels 211, 212, 214 and 215.

Pixel 211 is a horizontal neighbor of pixel 212, and is a vertical neighbor of pixel 214, and is a diagonal neighbor of pixel 215.

[00131] In illustrative implementations, the pixels in each superpixel are sufficiently small and sufficiently close together that light incident on the photodiodes of the pixels in the superpixel is locally homogeneous - that is, within a single superpixel, spatial variation in intensity of incident light is not substantial.

[00132] (However, in some cases: (a) the image sensor itself includes spatial light modulators—such as an LCD— for selectively modulating the intensity of incoming light; and (b) the light incident on the photodiodes of the superpixel is locally homogeneous if the effect of the spatial light modulators is subtracted, but is not locally homogeneous if the effect of the spatial light modulators is taken into account.)

[00133] Depending on the particular implementation, the spatial distribution of pixel sets within an image sensor may vary. In many cases, it is preferable for the pixel sets to be spatially distributed in a repeating pattern. For example, in some cases, 9 pixel sets are used in an image sensor, and each superpixel is a 3x3 pixel square, which is an easily repeatable pattern. For example, in Figure 2, the following spatial pattern of 9 pixel sets occurs in superpixel 251 : pixels from pixel sets 1, 2, 3 are in the top row of the superpixel, pixels for pixel sets 4, 5, 6 are in the middle row of the superpixel, and pixels for pixel sets 7, 8, 9 are in the bottom row of the superpixel. This spatial pattern repeats in superpixels 252, 253, 254. Or, for example, in some cases, a 3-pixel superpixel in an "L" shape is used: stacking an upright "L" and a bottom-up "L" gives a rectangle and thus produces a repeatable pattern.

[00134] Depending on the particular implementation, any shape of superpixels may be used. For example, in some cases, triangular or hexagonal superpixels are used. In some cases, the superpixels has a linear shape, such as horizontal (e.g., [12345][12345][12345]), and vertical (e.g., [12345] ^T [12345] ^T [12345] ^T ) However, in some cases, using a line-shaped superpixel has a disadvantage: if all "l"s are in one column; this tends in some cases to at least slightly increase the decoding error rate.

[00135] In some cases, the position of pixels from different pixel sets (within each superpixel) is rearranged from superpixel to superpixel (e.g. using a few fixed patterns, completely randomizing the order, or using a mathematical permutation).

[00136] For example, in some implementations, the spatial distribution of pixels from different sets of pixel sets within each superpixel is randomized in design time, then the random distribution is hardwired. For example, in some cases, randomization produces, for three line-shaped superpixels, a spatial arrangement of pixels from different pixel sets of [13452][35214][41253], rather than [12345][12345][12345]. Likewise, randomization of position of pixels from different pixel sets is applicable to any shape of superpixel, including square, rectangular, L-shaped, triangular or hexagonal.

[00137] In some cases, a multiplexer rearranges the shape or size of superpixels, such that superpixels with different sizes or shapes used at different times (e.g., for different applications, different frames, or at different times within a single frame) or at different regions within a single image sensor. For example, in some cases, a multiplexer reconfigures different superpixel shapes, e.g., 3x3, 4x3, or another shape. For example, in some implementations with CMOS pixels, a multiplexer is placed outside the pixel matrix area and directs separate TX/PD RST control signals to the pixels. Likewise, in some cases, a multiplexer rearranges the position of pixels from different pixel sets within one or more of the superpixels.

[00138] As noted above, in some cases, the spatial distribution of different sets of pixels changes from frame to frame. Alternatively, in some cases, the spatial distribution is fixed because of wiring of the pixel circuit.

[00139] In many implementations, the different sets of pixels in the image sensor are disjoint (i.e., no pixel belongs to more than one of the sets of pixels). Alternatively, different sets of pixels are overlapping. In some cases, a single pixel that is an element of two pixel sets A and B is subjected to a pixel modulation signal that comprises a superposition of the modulation signals for pixel sets A and B.

[00140] The size of the superpixel depends on the application. If, for example, a 9 pixel superpixel is not enough for a specific application and a 16 pixel superpixel is too much, a 12 pixel superpixel (4x3 or 3x4) may be used for that specific application. More pixel sets in a superpixel allows for more complicated coding strategies or code sharing. For example, a modulation signal comprising 18-bit gray code may be applied to the pixels of a 6x3 pixel superpixel.

[00141] In some use scenarios, spatial compression is achieved by treating each large superpixel as if it comprises smaller superpixels, and then applying modulation signals as follows: Modulation signals corresponding to lower bits of a code are applied to only one pixel per large superpixel, but modulation signals corresponding to higher bits of the code are applied to one pixel per smaller superpixel. For example, in some use scenarios, a 6x3 pixel superpixel is treated as if comprising two 3x3 pixel superpixels. In these use scenarios, a modulation pattern corresponding to each of the 4 lower bits of the 11-bit gray code is applied to just one pixel in the 6x3 superpixel, while modulation patterns corresponding to each of the 7 higher bits of the 11-bit gray code are applied to one pixel in each of the 3x3 superpixels. Spatial compression (by applying lower bits and higher bits in this manner) has many practical advantages. For example, in some 3D depth sensing applications, this spatial compression allows for using multiple higher bit modulation signals to determine, for each large superpixel for each frame, the 3D coordinates for multiple points in a scene. For example, in some implementations that use this spatial compression, a computer calculates 3D coordinates of four scene points, for each frame for each 6x3 superpixel. [00142] In some cases, the actual spatial distribution of pixel sets is hardwired in a fixed pattern. However, even then, in some cases, software is used to effectively vary the spatial distribution of pixel sets from frame to frame, by changing, from frame to frame, which bits of a modulation code are applied to which pixel set.

[00143] For example, in some cases of superpixel modulation: (a) in a first frame, software causes the x-th bit of a modulation signal to be applied to modulate pixel set x; and (b) in a second frame, software causes the x-th bit of a modulation signal to be applied to modulate pixel set y. For example, in some use scenarios: (a) in a first frame, the modulation signal for the first bit of a code is applied to a first pixel set, the modulation signal for the second bit of the code is applied to the second pixel set, and so on, and (b) in a second frame, the first bit of a code is applied to the second pixel set, the modulation signal for the second bit of the code is applied to the third pixel set, and so on. Thus, in this approach, the permutation of assignments (from code bits to pixel sets) varies from frame to frame. In some cases, the same permutation of assignments (of code bits to pixel sets) is used in all frames. In other cases, different permutations of assignments (of code bits to pixel sets) are applied in different frames in a fixed order. In yet other cases, a different random

permutation of assignments (of code bits to pixel sets) is used in each frame.

[00144] Here is a non-limiting example of varying the permutation of assignments (of code bits to pixel sets), in superpixel modulation. For a 5-bit code, there are 5 ! (factorial) different permutations of assignments of code bits to pixel sets. Different permutations are used in different frames. For example, (a) in a first frame, a first permutation is used, which maps bit 1 to pixel set 1, bit 2 to pixel set 2, bit 3 to pixel set 3, bit 4 to pixel set 4, and bit 5 to pixel set 5; and (b) in a second frame, a second permutation is used, which maps bit 1 to pixel set 4, bit 2 to pixel set 2, bit 3 to pixel set 5, bit 4 to pixel set 3, and bit 5 to pixel set 1.

[00145] In illustrative implementations, each superpixel comprises multiple imaging pixels. Each pixel in the superpixel is a horizontal, vertical or diagonal neighbor of at least one other pixel in the superpixel. In the example shown in Figure 2, pixel 215 has only two vertical neighbors (pixels 212, 217), only two horizontal neighbors (pixels 214, 216), and only four diagonal neighbors (pixels 211, 213, 217, 219).

[00146] In many cases, the pixels of a given pixel set are not horizontal, vertical or diagonal neighbors of each other. For example, in Figure 2, pixels 216 and 226 are each members of the sixth pixel set. None of the neighboring pixels (i.e. horizontal, vertical or diagonal neighbors) of pixel 216 is a member of the sixth pixel set. Likewise, none of the neighboring pixels (i.e. horizontal, vertical or diagonal neighbors) of pixel 226 is a member of the sixth pixel set.

Superpixel Modulation: Signals

[00147] Figure 3 is a diagram that shows five different signals for superpixel modulation that are simultaneously applied to five different pixel sets during a single frame. In Figure 3, the five signals 311, 312, 313, 314, 315 together comprise a binary 5 -bit gray code. The code is binary because each component of the code has only two allowed values. In Figure 3, the two allowed values are called "open" and "closed". However, the two allowed values may be called by any name, such as on and off, high and low, or 0 and 1.

[00148] In Figure 3, each of the five signals 311, 312, 313, 314, 315 for superpixel modulation corresponds to a bit of a 5-bit gray code. The 5-bit gray code has 2 ⁵ (i.e., 32) permutations of open and closed states of the 5 component signals.

[00149] In Figure 3, all 2 ⁵ (i.e., 32) permutations of open and closed states of the 5 component signals are applied to the pixel sets, during a single frame of the camera. For a 5- bit code (such as that shown in Figure 3), a single frame of the camera has 2 ⁵ (i.e., 32) subintervals, which are sometimes called "timeslots". Each timeslot corresponds to a single permutation of the code. For example, during timeslot 321 in Figure 3, the permutation is signal 315 closed, signal 314 closed, signal 313 open, signal 312 open and signal 311 closed. In contrast, during timeslot 323 in Figure 3, the permutation is signal 315 open, signal 314 open, signal 313 open, signal 312 open and signal 311 closed.

[00150] In Figure 3, a unique permutation (of binary states of the 5 component signals of the 5-bit gray code) exists for each of the 32 timeslots. Thus, each permutation (of states of the 5 component signals of the 5-bit gray code) is effectively a "timestamp" that uniquely identifies a timeslot within a frame.

[00151] In some implementations of this invention, a 9-bit binary gray code is employed for superpixel modulation. The nine bits of this code are represented by 9 modulation signals. The 9-bit code has 2 ⁹ (i.e., 512) permutations of open and closed states of the 9 modulation signals. All 512 permutations are applied to pixels in each frame of the camera. Thus, for this 9-bit code, each frame has 512 subintervals, sometimes called timeslots. A unique permutation of the 9-bit code (and thus of the open and closed states of the 9 modulation signals) exists for each of the 512 timeslots. In some cases in which a 9-bit code is used: (a) there are nine pixel sets; (b) the 9 modulation signals are applied to the 9 pixel sets, one signal per pixel set; and (c) each superpixel has 9 pixels, one pixel per pixel sets. Thus, the 9 modulation signals are applied to the 9 pixels in a superpixel, one signal per pixel.

[00152] Gray codes are robust against noise, which is advantageous.

[00153] Depending on the particular implementation of this invention, the modulation signals for superpixel modulation may be either (a) continuous time (analog), (b) discrete- time, or (c) digital (discrete in both amplitude and time).

[00154] For example, in many examples of this invention, digital modulation signals are used for superpixel modulation. In that case, the amplitude of the modulation signal is treated as being in one of two binary states (e.g., 0 or 1), and the signal changes only at discrete time intervals.

[00155] Alternatively, modulation signals for superpixel are treated as continuous in amplitude. For example, in some cases, a computer normalizes the range of continuous values to an interval [0,1]. In that case, for example, a value of 0.241 may mean that 24.1% of electrons are moved to storage.

[00156] There is no limitation of when the signal change can occur in time. In some implementations, the signal changes continuously in time.

[00157] In some cases, analog modulation signals are used for superpixel modulation, which analog signals are treated as continuous in both amplitude and time.

[00158] In some implementations, the modulation signals for superpixel modulation are non-periodic. An example of such an aperiodic signal is 00110010101101000011010, which does not have one specific frequency or phase. In other cases, the modulation signals are periodic. Or, in some cases, a periodic signal such as 01010101010101 is applied to a first pixel set, and a second periodic or aperiodic signal (e.g., periodic signal

10000100001000) is applied to a second pixel set.

[00159] In some cases, the frequency, phase, or amplitude of the modulation signal for a pixel set changes over the duration of one frame. In other cases, the frequency, phase, or amplitude of the modulation signal for a pixel set is constant within each single frame, but changes from frame to frame.

[00160] In illustrative implementations, each modulation signal is applied to only one pixel in each superpixel. Alternatively, a single modulation signal is applied to multiple pixels within a single superpixel. [00161] In illustrative implementations: (a) all pixels which belong to the same pixel set have the same modulation signal; and (b) different pixel sets receive different modulation signals.

[00162] In superpixel modulation, each pixel set receives its own, separate modulation signal.

[00163] In many use scenarios, the amplitude of each modulation signal for superpixel modulation varies as a function of time during a single frame. However, in other use scenarios: (a) the amplitude or phase of a modulation signal is constant over time within a single frame, where the constant is different for different pixel sets, or (b) the amplitude or phase of a modulation signal is the same constant for different pixel sets. In other words, this invention is not limited to signals that change over time or that are different among different sets of pixels.

[00164] In some cases, the modulation functions for pixel sets changes frame to frame.

[00165] Depending on the particular implementation, the manner in which the modulation signals for superpixel modulation are physically implemented may vary. For example, in some cases, the pixels are CMOS imaging pixels, an electrical modulation signal causes a charge transfer (TX) transistor to transfer charge from a photodiode to a storage device (e.g., floating diffusion). In some cases: (a) each pixel has only one photodiode and more than more than one TX transistor; (b) more than one storage device (e.g., floating diffusion) is dedicated to only that pixel; and (c) each of the TX transistors in that pixel controls transfer to a different storage device.

[00166] In other cases, an electrical modulation signal causes electrodes to vary the position of mirrors in a DMD (digital micromirror device) by elecrostatic attraction, thereby causing the DMD to selectively reflect light towards some pixels in the image sensor plane, and to reflect light away from some other pixels in the image sensor plane.

[00167] In other cases, an electrical modulation signal causes the properties of an SLM (spatial light modulator), such that the SLM selectively attenuates light before it reaches the image sensor plane.

[00168] Physical implementation of modulation signals is discussed in more detail below. Effect of Superpixel Modulation

[00169] In some implementations, superpixel modulation controls the response of a pixel to incident light.

[00170] Figure 4 is a conceptual diagram, illustrating how signals for superpixel modulation control the response of pixels to incident light.

[00171] Figure 4 shows the effect of superpixel modulation. For ease of illustration (to show how superpixel modulation works), Figure 4 does not show the combined effect of both superpixel modulation and curtain modulation.

[00172] In Figure 4, plot 400 is a plot of I _xy (t) 401 versus time. I _xy (t) is the intensity of light incident on the x, y ^tft superpixel (i.e., the superpixel which has coordinates x, y on the sensor plane) as a function of time. The time interval shown in plot 400 is a complete camera frame, from the start of the frame to the end of the frame, (i.e., t^ _{rame start} to tframe_en _d )- The intensity of incident light due to ambient background illumination has a constant amplitude of a (402). A small peak of intensity occurs at time 404 due to interreflections from the scene. The highest peak of intensity 406 occurs at time t _hit . For example, in some cases: (a) a line of laser light sweeps across the scene during each frame, and (b) the sweeping line of laser light reaches a point in the scene and reflects, such that the reflected light reaches the image sensor at time t _hit .

[00173] In Figure 4, superpixel modulation functions G ₁ (t)to G _n (t) 403 are applied to the n pixel sets in the superpixel, one modulation function per pixel set. In the example shown in Figure 4: (a) n = 5; (b) a 5-bit gray code is employed; (c) G ₁ (t) to G _n (t)are the five signals that comprise the 5 -bit gray code, one signal per bit of the code, as illustrated in Figure 3.

[00174] In Figure 4, Vi(t) is a value for the i ^th pixel set in the superpixel. Specifically, Vi(t) is equal to the integral of the product of l _Xi y(t) and Gj(t) (the i ^th superpixel modulation function). This integral Vi(t) is equal to the area of dark region under the curve in plots 408, 410, 412. In plots 408, 410, 412, the curve that defines the top of the dark region is equal to the product of the product of l _Xi y(t) and G^t).

[00175] In plot 408, superpixel modulation signal G ₁ (t) is signal 315 shown in Figure 3. During the first half of the frame (shown in the left side of plot 408), signal 315 is closed (i.e., equal to 0), and thus the product of l _Xi y(t) and G-^it) is equal to zero, and there is no dark region under the curve. During the second half of the frame (shown in the right side of plot 408), signal 315 is open (i.e., equal to 1), and the product of l _xy (t) and G^t) is equal to / _y (t), and there is a dark region under the curve.

[00176] In plot 410, superpixel modulation signal G ₂ (t) is signal 314 shown in Figure 3. During the first and fourth quarters of the frame, signal 314 is closed (i.e., equal to 0), and thus the product of l _xy (t) and G ₂ (t) is equal to zero, and there is no dark region under the curve. During the second and third quarters of the frame, signal 314 is open (i.e., equal to 1), and the product of l _xy (t) and G ₂ (t) is equal to of I _xy (t), and there is a dark region under the curve.

[00177] In plot 412, superpixel modulation signal G _n (t) is G ₅ (t), which is signal 311 shown in Figure 3. The frame is divided into 32 timeslots. During timeslots in which signal 311 is closed (i.e., equal to 0), the product of l _xy (t) and G _s (t) is equal to zero, and there is no dark region under the curve. During timeslots in which signal 311 is open (i.e., equal to 1), and the product of l _xy (t) and G ₅ (t) is equal to of I _xy (t), and there is a dark region under the curve.

[00178] In Figure 4, the incident light function is identical for each of the pixel sets of the superpixel (that is, l _xy (t) is the same for all of the pixels in a superpixel, due to their close proximity). However separate superpixel modulation signals G ₁ (t)to G _n (t) are applied to the n pixel sets in the superpixel. Thus, each of the pixel sets in the superpixel receives a separate superpixel modulation signal.

[00179] A computer calculates Vi(t) (the integral of the product of I _xy (t) and G^t)) for each pixel set in the superpixel 405. For example, a computer calculates Vi(t) for the first pixel set which received superpixel modulation signal G-^t), and calculates v ₂ (t) for the second pixel set which received superpixel modulation signal G ₂ (t) , et cetera.

[00180] A computer also calculates a threshold 413. In some cases, the threshold 413 is equal to (max Vi— min v )/2 , where i = 1,2, ... , n, and max Vi is the maximum Vi(t) for all the pixels in the superpixel, and min v _t is the minumum v _t (t) for all the pixels in the superpixel.

[00181] A computer then calculates code C _xy 409 for the x, y ^tft superpixel (i.e., the superpixel which has coordinates x, y on the sensor plane). Code C _xy has the same number of bits as the number of pixel sets in the superpixel. A computer sets the i ^th bit of code C _xy to 1 if the Vi(t) is greater than the threshold; otherwise the computer sets the i ^th bit of code C _xy to 0. In the example shown in Figure 4, the computer: (a) sets the first bit of code C _xy to 1 because i½(t) is greater than the threshold; (b) sets the n ^th (in this example, the 5th) bit of code C _{x y} to 0 because v _n (t) is less than the threshold.

[00182] Each permutation of code C _{x y} corresponds to a unique timeslot (time interval) within a single frame. This unique timeslot is the timeslot within which time t _hit occurred. Thus, each permutation of code C _{x y} is effectively a timestamp that uniquely identifies a timeslot (time interval) within a single camera frame, such that incident light striking the pixel reached its maximum peak during this timeslot.

[00183] In some 3D depth-sensing implementations of this invention, a galvanometer mirror rotates, causing a plane of laser light to sweep across a scene during each camera frame. The angle of the mirror (and thus the angle of the plane of laser light) is a function of time within the frame. Thus, each permutation of code C _xy corresponds to a unique timeslot (time interval) within a single camera frame, which in turn corresponds to a unique angle of the plane of laser light. 411. A computer uses this angle, and other factors determined during calibration, to compute the 3D position of the scene point that reflected the laser light to the superpixel. The computer stores, for each superpixel and each angle, the 3D position of the corresponding scene point. During subsequent operation of a depth sensor, a computer calculates a permutation of code C _{x y} for each superpixel, as described above, and then uses the lookup table to compute the 3D position of the scene point that corresponds to that scene point.

[00184] In some implementations, each pixel stores charge to only one storage device, which storage device is dedicated to that pixel and does not store charge for any other pixels. When a pixel has only one storage device, less than all of the incident illumination is measured.

[00185] In the example shown in Figure 4, each pixel stores charge to only one storage device, which storage device is dedicated to that pixel and does not store charge for any other pixels. Thus, Figure 4 illustrates a single storage device implementation.

[00186] In single storage device implementations, for each camera frame, the total photodiode charge measured by each pixel in the i ^th pixel set in a superpixel is equal to the value of v _t (t) at the end of the frame (i.e., is equal to Vi(tf _{rame end} ) . This value, in turn, depends on the superpixel modulation function G _n (t) that is applied to the i ^th pixel set. Thus, the superpixel modulation function Gj(t) determines the response of the pixels in the i ^th pixel set to incident light. (In most cases, each superpixel includes only one pixel from each pixel set. In those cases, the i ^th pixel set is simply the i ^th pixel.) [00187] Recall that, for ease of illustration, Figure 4 shows only superpixel

modulation, and does not show curtain modulation. However, in many implementations of this invention, the value of v _t (t) at the end of the frame depends on both the superpixel modulation signal and the curtain modulation signal.

Curtain Modulation and Superpixel Modulation, Applied Together

[00188] In illustrative implementations, both current modulation and superpixel modulation are simultaneously applied to each pixel of the image sensor.

[00189] In illustrative implementations, whether a particular pixel is sensitive to light at any given time is controlled by the logical AND of the curtain and superpixel modulation signals. That is, a pixel is sensitive to light at a given time if, at the given time, both (a) the curtain modulation signal for that pixel is high and (b) the superpixel modulation signal for that pixel is high. The pixel is not sensitive to light when one or both of the curtain and modulation signals for that pixel is low. Furthermore, in illustrative implementations: (a) the curtain modulation signal for a pixel is high when the pixel is in the active region, and not at other times; and thus (b) a pixel is sensitive to light when the pixel is in the active region of the image sensor and, simultaneously, the superpixel modulation signal for the pixel is high.

[00190] In illustrative implementations: (a) a curtain modulation signal for a given pixel is applied to the entire row (or column) of pixels in which the given pixel is located; and (b) a superpixel modulation signal for a given pixel is applied to the entire pixel set of which the given pixel is a member.

[00191] In some implementations, the sensitivity of the pixel to light may approximate a binary state (sensitive / not sensitive) which is encoded by the logical AND of the curtain and superpixel modulation signals.

[00192] As used herein, unless the context clearly indicates otherwise, "high" and "low" are Boolean logic values. For example, "high" and "low" Boolean logic values may be physically represented by particular voltage ranges. Which voltage range is assigned to which Boolean logic value is arbitrary. Thus, for example: (a) the Boolean logic value "high" may be represented by a first voltage range; (b) the Boolean logic value "low" may be represented by a second voltage range; and (c) voltages in the first voltage range may be greater than voltages in the second voltage range, or vice versa.

[00193] In illustrative implementations, curtain and superpixel modulation are simultaneously applied to the image sensor as follows: In curtain modulation, an active region repeatedly sweeps across the image sensor during a single camera frame. Thus, the active region repeatedly moves through each superpixel during a single camera frame.

Simultaneously, throughout each camera frame, superpixel modulation signals are applied to each given superpixel in the image sensor, including: (a) during many short periods (in each camera frame) in which the given superpixel is in the active region of the image sensor; and (b) during the many short periods (in each camera frame) in which the given superpixel is not in the active region of the image sensor. The moving active region is narrow compared to the image sensor as a whole. Thus, a given superpixel is in in the active region during only a small percentage of the total time of a camera frame.

[00194] In illustrative implementations, when both superpixel and curtain modulations are performed simultaneously, all of the bits of the code encoded by superpixel modulation are presented during each camera frame. More specifically, in illustrative implementations: (a) the superpixel modulation signals that are applied to n different pixel sets together comprise an /7-bit binary code; (b) each bit of the code corresponds to a different time interval during a single frame of the camera; (c) the amount of time that it takes to present all n bits of the n bit code equals the amount of time that elapses in single frame of the camera; and (d) all of the bits of the code occur in a single frame of the camera.

[00195] In illustrative implementations, when both superpixel and curtain modulations are performed simultaneously, the dot of light for curtain modulation traverses the same path in each camera frame. Thus, in each camera frame, the dot of light starts at the same starting point, traverses the same path, and ends at the same ending point. As the laser light traverses this path during each camera frame, the laser light illuminates different parts of the scene at different times during each frame, such that each part of the scene is illuminated at least once during each frame. For example: (a) the path that the dot of light travels during a single camera frame may comprise moving vertically up repeatedly, such that, during the single frame, the dot of light traverses an entire 2D spatial region by traveling up different vertical "columns" of the region; (b) the path may start at the bottom right corner of the 2D spatial region, repeatedly go up different vertical "columns" of the region, and end at the upper left corner of the region; and (c) during portions of the path, the dot of light may be "off (put differently, the path may have gaps that occur when the dot of light is off).

[00196] In illustrative implementations, the movement of the dot of light for curtain modulation may be achieved by rotating a galvo mirror. This rotation may cause a laser beam to deflect in different directions and to illuminate a scene with a moving dot of laser light. For example, the galvo mirror may cause a mirror to rotate by 40 degrees during each vertical upward move of the dot of light. As the dot of light moves from the top of one "column" to the bottom of the adjacent "column", the dot of light may be off.

[00197] In some cases, both curtain and superpixel modulation are achieved by controlling storage of electric charge. In those cases, a particular pixel may be sensitive to light for a given time period only if the combined effect of the curtain and superpixel modulation signals is such that electric charge accumulated in the photodiode for that time period is transferred to storage (e.g., floating diffusion) and is not flushed.

[00198] In some cases, both curtain and superpixel modulation are achieved by one or more SLMs (e.g., reflective or transmissive SLMs) selectively attenuating light. In these cases, a particular pixel may be sensitive to light for a given time period only if, during that time period, the combined effect of the curtain and superpixel modulation signals is such that light from the scene reaches the pixel and is not blocked (or deflected away) by one or more of the SLMs.

[00199] In illustrative implementations, the frequency of curtain modulation is greater than the frequency of superpixel modulation. For example, during each period of a superpixel modulation signal, the dot of light for curtain modulation may move up many vertical "columns" of the scene.

[00200] The following seventeen paragraphs describe a non-limiting example of this invention. In this example, which the frequency of curtain modulation is greater than the frequency of superpixel modulation.

[00201] In this example, the image sensor comprises 1998 x 1500 pixels, that is 1998 columns of pixels and 1500 rows of pixels. There are nine pixel sets. Each superpixel comprises 9 pixels, arranged in a 3x3 matrix. Each superpixel contains one pixel from each of the pixel sets. The image sensor comprises 666 x 500 superpixels.

[00202] In this example, the dot of light goes up 1998 times during each camera frame (each vertical movement up may be considered as movement in a vertical "column").

[00203] In this example, the dot of light displaces horizontally from the right to the left, after each vertical movement up. There are 1998 up movements of the dot of light during each camera frame. The dot of light is off during each down movement, as the dot of light moves from the top of one column to the bottom of another column of the image sensor. Each up movement is shifted by 1/1998 of the horizontal range, relative to the previous up movement (the horizontal range being the range of horizontal movement of the dot of light during a single camera frame). In this example, the horizontal range is 50 degrees. Thus, each horizontal displacement of the dot of light, from one vertical column to the adjacent vertical column while the dot of light is off, is achieved by a galvo mirror rotating

approximately 1.5 arcminutes. Each vertical movement up of the dot of light, while the dot of light is on, is achieved by a galvo mirror rotating approximately 40 degrees (i.e., 2,400 arcminutes). Thus, the mirror for vertical movement of the dot of light is rotating much faster than the mirror for the horizontal movement of the dot of light.

[00204] In this example, the superpixel modulation signals encode a 9-bit binary code. There are 9 separate superpixel modulation signals, one for each of the 9 pixel sets.

[00205] In this example, during a single frame of the camera: (a) the superpixel modulation signal for pixel set #1 goes high-and-low 1 time; (b) the superpixel modulation signal for pixel set #2 goes high-and-low 2 times; (c) The superpixel modulation signal for pixel set #3 goes high-and-low 4 times; (d) the superpixel modulation signal for pixel set #4 goes high-and-low 8 times; (e) the superpixel modulation signal for pixel set #5 goes high- and-low 16 times; (f) the superpixel modulation signal for pixel set #6 goes high-and-low 32 times; (g) the superpixel modulation signal for pixel set #1 goes high-and-low 64 times; (h) the superpixel modulation signal for pixel set #8 goes high-and-low 128 times; and (i) the superpixel modulation signal for pixel set #9 goes high-and-low 256 times. Each time that a superpixel modulation signal goes high and then low is a period of that signal. Thus, in this example: (a) the superpixel modulation signal for pixel set 1 has one period in each camera frame; (b) the superpixel modulation signal for pixel set 5 has 16 periods in each camera frame; and (c) the superpixel modulation signal for pixel set 9 has 256 periods in each camera frame.

[00206] In this example, each superpixel modulation signal is a square wave with a duty cycle of 50%. That is, in this example, the proportion of ON and OFF in each superpixel modulation signal is 50:50.

[00207] In this example, the curtain window width is 9 pixels (3 superpixels).

[00208] In this example, the curtain modulation signal for each row of pixels goes high-and-low 1998 times during one camera frame. During each period of the curtain modulation signal, the signal goes high and then low. For each row of pixels, the curtain modulation signal is high approximately 9/1998 = 0.45% of the total time of each camera frame. (The curtain modulation signal is high at different times for different rows of pixels.). For approximately 1989/1998= 99.65% of each camera frame, the curtain modulation signal for a given row of pixels is low.

[00209] Thus, in this example, for a given row of pixels, the curtain modulation signal will be low most of the time, and will be high during 1998 very short peaks in each camera frame. These peaks are regularly spaced in time throughout the total duration of the camera frame.

[00210] In this example, for the pixel from pixel set #9 in a given superpixel, there will be approximately 7.8 peaks of curtain modulation during one period of superpixel modulation.

[00211] In this example, the TX gate of a given pixel is high approximately 0.45% * 50% = 0.225%) of the time during a camera frame.

[00212] In this example, 1998 periods of the curtain modulation signal for a given pixel occur during a single camera frame. The framerate of the camera is 30 fps (frames per second). Each camera frame has a duration of approximately 33ms.

[00213] Thus, in this example, the curtain modulation signal for a given pixel has a frequency of approximately 60 kHz. In this example, the curtain modulation signal is a pulse wave signal with proportion of 9: 1989 ON:OFF. The length of one ON pulse will be approx. 75 nanoseconds.

[00214] In this example, the highest frequency superpixel modulation is the superpixel modulation for pixel set #9. The superpixel modulation signal for pixel set #9 has 256 periods during each camera frame - that is, 7680 periods per second. The superpixel modulation signal is a square wave with a 50% duty cycle (50:50 ON:OFF).

[00215] Thus, in this example, the fastest superpixel modulation signal has a frequency of approximately 7.6 kHz and the curtain modulation signal has a frequency of approximately 60 kHz. The latter signal has a higher frequency than the former.

[00216] In this example, each pixel will be sensitive to light only when both the curtain and superpixel modulation signals for that pixel are high. (This is because, in this example, whether a given pixel is sensitive to light is a binary state that is the logical AND of the curtain and modulation signals.).

[00217] In this example, the superpixel modulation signal is high during 50% of each camera frame and the curtain modulation signal is high during approximately 0.45% of each camera frame. In this example, a given pixel is sensitive to light in many small periods of time during each camera frame. In this example, these small periods of time in each camera frame are (taken together) equal to approximately (0.5 x 0.0045)% = 0.00225% of the total time of each camera frame.

[00218] The example described in the preceding seventeen paragraphs is non-limiting. This invention may be implemented in many other ways.

[00219] In the above example, the number of vertical columns of movement of the dot of light during each frame is equal to the number of columns of pixels in the image sensor. However, the number of vertical columns of movement does not have to be equal to the number of columns of pixels in the image sensor. Alternatively, the number of vertical columns of movement may be lower (or higher) than the number of columns of pixels in the image sensor. In some cases, the number of columns of vertical movement is evenly divisible by the number of columns of pixels (or superpixels), or vice versa. In other cases, these numbers are not evenly divisible by each other, or are co-prime with each other.

[00220] Increasing the frequency of the curtain modulation signal - and thus increasing the number of vertical (or horizontal) movements of the dot of light - has both advantages and disadvantages. Increasing the frequency improves the S R and the resolution of the scan, but may require a faster transistor switching speed and may make synchronization more difficult.

[00221] In illustrative implementations, each pixel in the active region is sensitive to light during portion(s) of each period of the superpixel modulation signal and is not sensitive to light during other portion(s) of each period of the superpixel modulation signal. At many times during each camera frame, less than all of the pixels in each superpixel are sensitive to light - even in the active region of the image sensor.

[00222] Figure 40 shows an example of curtain and superpixel modulation signals that control a pixel's sensitivity to light. Figure 40 shows three signals for the pixel: a superpixel modulation signal 4001, a curtain modulation signal 4010, and a signal 4020 that controls the pixel's time-varying sensitivity to light (the "light sensitivity signal").

[00223] In the example shown in Figure 40, the superpixel modulation signal 4001 for the pixel is a square wave with a 50% duty cycle. Each period of the superpixel modulation signal 4001 consists of the signal going high then low. For example, period 4002 of the superpixel modulation signal 4001 starts at time 4032 and ends at time 4037. The same superpixel modulation signal 4001 is applied to all pixels in the pixel set of which the pixel is a member. [00224] In the example shown in Figure 40, the curtain modulation signal for the pixel is a pulse wave with a 0.45% duty cycle (i.e., the signal is high 0.45% of each period and low 99.55%) of each period). Each period of the curtain modulation signal 4010 consists of a short pulse (in which the signal goes high) followed by a much longer period in which the signal is low. For example, period 4012 of the curtain modulation signal 4010 starts at time 4033 and ends at time 4034. The same curtain modulation signal 4010 is applied to the entire row of pixels, of which the pixel is a member.

[00225] In the example shown in Figure 40, a total of approximately 7.8 periods of the curtain modulation signal 4010 occur in each period of the superpixel modulation signal 4001.

[00226] In the example shown in Figure 40, the light sensitivity signal 4020 (which controls the pixel's sensitivity to light) is the logical AND of the pulsed curtain modulation signal 4010 and the square-wave superpixel modulation signal 4001. Thus, the light sensitivity signal 4020 is high only when the curtain modulation signal 4010 and the superpixel modulation signal 4001 are both simultaneously high. For example, pulse 4052 of the light sensitivity signal 4020 occurs at time 4033, when pulse 4042 of the curtain modulation signal 4010 occurs and the superpixel modulation signal 4001 is high. Likewise, pulse 4053 of the light sensitivity signal 4020 occurs at time 4034, when pulse 4043 of the curtain modulation signal 4010 occurs and the superpixel modulation signal 4001 is high. A pulse of the light sensitivity signal 4020 does not occur at time 4036, because superpixel modulation signal 4001 is low then.

[00227] In illustrative implementations, the rising and falling edges of the superpixel modulation signal 4001 are not synchronized with the pulses of the curtain modulation signal 4010. For example, in Figure 40, the rising and falling edges of the superpixel modulation signal 4001 occur at different times (e.g., 4032, 4035, 4037) than do the pulses (e.g., 4042, 4043, 4044) of the curtain modulation signal 4010.

[00228] Figure 41 is a flowchart that shows a method for curtain modulation and superpixel modulation. In Figure 41, both of the modulations are achieved by controlling the storage of electric charge that accumulates in the photodiode. In the example shown in Figure 41, the method includes the following steps: Light from a scene in incident on the photodiode throughout the entire method (Step 4101). A curtain modulation signal is applied to the pixel. If the curtain modulation signal is high, then the pixel is in the active region of the image sensor. If the curtain modulation signal is low, then the pixel is not in the active region (Step 4102). If the pixel is not in the active region, then charge in the photodiode is not transferred to floating diffusion (FD) storage (Step 4104). A superpixel modulation signal is applied to the pixel (Step 4103). If the pixel is in the active region and the superpixel modulation signal is high, then charge is transferred from the photodiode to FD storage (Step 4105). If the superpixel modulation signal is low, then charge is flushed from the photodiode and is not stored in FD storage (Step 4106).

[00229] Figure 42 is another flowchart that shows a method for curtain modulation and superpixel modulation. In the example shown in Figure 42, the method includes the following steps: Set FPS (frames per second) and other timing parameters (Step 4201). An electronic clock sends clock signals. For example, the electronic clock may include an electronic oscillator regulated by a quartz crystal (Step 4202). Start frame (Step 4203).

Output control signals to control the camera subsystem and the illumination subsystem. In some implementations, the camera subsystem includes an SLM (such as a DMD or LCD) for selectively attenuating light before the light reaches the image sensor. The control signals that control the camera subsystem may include curtain modulation signals and superpixel modulation signals. The control signals that control the illumination system: (a) may include signals that turn a light source on and off (either by causing the light source to not emit light or by occluding light from the light source); and (b) may include signals that control galvos (i.e., galvanometers that move mirrors to different positions to vary the angle at which laser light is reflected from the mirrors.) (Step 4204). Receive data from camera (Step 4205). Process a camera frame (Step 4206), taking into account calibration data that was previously acquired (Step 4208). Output resulting data (Step 4207).

Modulation of Storage

[00230] In some implementations, signals control storage of charge in a CMOS pixel for superpixel modulation, or for curtain modulation, or for both.

[00231] Figures 5, 7 and 9 show three examples of CMOS circuits, in illustrative implementations of this invention. Figures 5, 7 and 9 are each a circuit schematic for a portion of a CMOS pixel. The CMOS pixel has one floating diffusion storage device.

[00232] COL means column selector 507.

[00233] FD means floating diffusion 509. The floating diffusion stores electric charge.

[00234] PD RST means photodiode reset transistor 513.

[00235] ROW means row selector 511. [00236] TX means gate transistor 503 in Figures 5 and 7.

[00237] TXi means gate transistor 523 in Figure 9.

[00238] TX ROW means row selector 505 for charge transfer.

[00239] PD RST ROW means photodiode row selector 521.

[00240] RST means floating diffusion reset transistor 517. RST causes floating diffusion 509 to be reset (i.e., the charge in floating diffusion 509 to be drained) when RST is high.

[00241] SF means source follower 515. The source follower is used during the readout.

[00242] V means voltage source.

[00243] In Figures 5 and 6, TX causes charge to be transferred from the photodiode 501 to floating diffusion 509 when voltages at COL and TX ROW are simultaneously high. In Figures 5 and 6, the TX ROW and COL signals that are applied to a pixel are

simultaneously high when the both of the following occur at the same time (i) the pixel is in the active region of the image sensor and (ii) the superpixel modulation for the pixel is high. In Figures 5 and 6, the time period in which TX ROW and COL are simultaneously high is a short pulse before PD RST goes high. (Recall that, in illustrative implementations: (a) a superpixel modulation signal is applied to an entire pixel set; (b) when a superpixel modulation signal is applied to a pixel, it is applied to the entire pixel set of which the pixel is a member; and (c) each superpixel includes multiple pixels, each from a different pixel set. For example, in some cases: (a) there are nine pixel sets and nine separate superpixel modulation signals; (b) each superpixel includes 9 pixels, one pixel from each of the pixel sets; (c) a separate modulation signal is applied to each of the pixel sets; and (d) thus 9 separate superpixel modulation signals are applied to the pixels in each superpixel, one for each pixel.)

[00244] In Figures 9 and 10, TXi causes charge to be transferred from the photodiode 501 to the storage FD 509 when TXi is high. In Figures 9 and 10, the TXi signal for a pixel is high when the pixel is in the active region of the image sensor and simultaneously the superpixel modulation for the pixel is high.

[00245] In Figures 7 and 8, PD RST causes photodiode 501 to be reset (that is, the charge in photodiode 501 to be drained) when COL and PD RST ROW are simultaneously high. In Figures 7 and 8, the COL and PD RST ROW signals that are applied to a pixel are simultaneously high at the following times (in addition to other times): (a) a short pulse when the pixel enters the active region of the image sensor; or (b) the entire time that the pixel is not in the active region of the image sensor.

[00246] In Figures 5, 6, 9 and 10, PD RST causes photodiode 501 to be reset (that is, the charge in photodiode 501 to be drained) when PD RST is high. In Figures 5, 6, 9 and 10, the PD RST signal that is applied to a pixel is high at the following times (in addition to other times): (a) at short pulse when the pixel enters the active region of the image sensor; or (b) the entire time that the pixel is not in the active region of the image sensor.

[00247] In Figures 5, 6, 7, 8, 9 and 10, readout occurs when voltages at COL and ROW are simultaneously high.

[00248] Figures 6, 8 and 10 are timing diagrams for Figures 5, 7 and 9, respectively. In Figures 6, 8 and 10, the entire exposure occurs during a single camera frame, and readout and floating diffusion reset occur after the frame. In Figures 6, 8 and 10, ModExp 531 is a modulation signal that is one of the components of the n-bit code. Certain other signals in these Figures are derived from ModExp, as discussed below.

[00249] At multiple times during the single camera frame, the then accumulated charge in photodiode 501 is transferred to floating diffusion 509 (i.e., each time that the voltages at COL and TX ROW are simultaneously high).

[00250] Also, at multiple other times during a single camera frame, the then accumulated charge in photodiode 501 is drained without being readout or measured (e.g., when PD RST is high during 551, 552, 553, 554, 555 in Figure 6, when both PD RST ROW and COL are high during 561, 562, 563 in Figure 8, and when PD RST is high during 571, 572, 573, 574, 575 in Figure 10).

[00251] Figure 11 A, Figure 1 IB and Figure 11 C are conceptual diagrams of a superpixel, in which each pixel has multiple storage devices. In Figure 11 A, the same transfer gates are employed for superpixel modulation and curtain modulation. In Figures 1 IB and 11C, different transfer gates are employed for superpixel modulation and for curtain modulation.

[00252] In Figures 11 A-l 1C, each pixel stores charge to multiple storage devices. Each of these multiple storage devices is dedicated to only that pixel and does not store charge for any other pixel. For example, in Figures 11 A-l 1C, pixel 1103 has a set of multiple storage devices 1125, and pixel 1105 has a non-overlapping set of multiple storage devices 1135. [00253] In the example shown in Figure 11 A, light from a scene 1100 strikes the photodiode (e.g., 1107, 1109) of each pixel (e.g., 1103, 1105) in a superpixel 1101. For each pixel, modulation signals cause the then accumulated photodiode charge in the pixel to be transferred to one of the storage devices for that pixel. Which storage device receives a particular transfer of charge is determined based on the modulation signal. For example, for pixel 1103, a modulation signal causes transfer gate 1111 to transfer charge from photodiode 1107 to storage device 1121 at some times during a single camera frame, and causes transfer gate 1113 to transfer charge from photodiode 1107 to storage device 1123 at some other times during the camera frame. Likewise, for pixel 1105, a different modulation signal causes transfer gate 1115 to transfer charge from photodiode 1109 to storage device 1131 at some times during a single camera frame, and causes transfer gate 1117 to transfer charge from photodiode 1109 to storage device 1133 at some other times during the camera frame.

[00254] In the example shown in Figure 11 A, a single level of transfer gates 1141 includes transfer gates 1111, 1113, 1115, 1117). This single level of hardware 1141 is controlled by the logical AND of the signals for superpixel modulation and curtain modulation.

[00255] Figures 1 IB and 11C are the same as Figure 11 A, except that separate hardware levels are employed for curtain modulation and superpixel modulation,

respectively. In Figures 1 IB and 11C, curtain modulation occurs at levels 1145 and 1146, respectively. Levels 1145 and 1146 comprise transfer gates (including transfer gates 1112, 1114, 1116, 1118). In Figures 1 IB and 11C, superpixel modulation occurs at levels 1143 and 1147, respectively. Levels 1143 and 1147 comprise transfer gates (including transfer gates 1111, 1113, 1115, 1117). For example, in Figures 1 IB and 11C, charge is transferred from photodiode 1107 to storage 1121 if and only if transfer gate 1111 (for superpixel modulation) and transfer gate 1112 (for curtain modulation) are both high. Thus, in Figure 1 IB, the effect of two hardware levels 1143, 1145, taken together is that storage of electric charge is controlled by the logical AND of the curtain modulation and superpixel modulation.

Likewise, in Figure 11C, the effect of two hardware levels 1146, 1147, taken together is that storage of electric charge is controlled by the logical AND of the curtain modulation and superpixel modulation. In Figures 11B and 11C, superpixel modulation and curtain modulation occur simultaneously. In Figure 1 IB, superpixel modulation occurs spatially closer to the photodiode than does curtain modulation. In Figure 11C, this spatial order is reversed: that is, curtain modulation occurs spatially closer to the photodiode than does superpixel modulation.

[00256] Figure 12 is a circuit schematic for a portion of a CMOS pixel that has three floating diffusion storage devices. In Figure 12:

[00257] COL means column selector 507.

[00258] FD1 is floating diffusion 549. FD2 is floating diffusion 559. FD3 is floating diffusion 569.

[00259] PD RST is photodiode reset transistor 513. In Figure 12, PD RST causes photodiode 501 to be reset (that is, the charge in photodiode 501 to be drained) when PD RST is high. In Figure 12, the PD RST signal that is applied to a pixel is high at the following times (in addition to other times): (a) a short pulse when the pixel enters the active region of the image sensor; or (b) the entire time that the pixel is not in the active region of the image sensor.

[00260] ROW is row selector 511.

[00261] RSTl is floating diffusion reset transistor 547. RST2 is floating diffusion reset transistor 557. RST3 is floating diffusion reset transistor 567. In Figure 12: (a) RSTl causes floating diffusion 509 to be reset when RSTl is high; (b) RST2 causes floating diffusion 509 to be reset when RST2 is high; and (c) RST3 causes floating diffusion 509 to be reset when RST3 is high. The reset causes charge in floating diffusion 509 to be drained. SF is source follower 515. In illustrative implementations, RSTl, RST2, RST3 are high only after readout, in order to reset FD1, FD2, FD3, respectively, to a "zero state" before the next frame begins.

[00262] TXil is transfer gate 543. TXi2 is transfer gate 553. TXi3 is transfer gate 563. In Figure 12: (a) TXil causes charge to be transferred from the photodiode 501 to the storage FD1 549 when TXil is high; (b) TXi2 causes charge to be transferred from the photodiode 501 to the storage FD2 559 when TXi2 is high; and (c) TXi3 causes charge to be transferred from the photodiode 501 to the storage FD3 569 when TXi3 is high.

[00263] In some implementations, signals control storage of charge for CCD pixels for superpixel modulation, or for curtain modulation, or for both.

[00264] Figure 13 is a conceptual diagram of CCD pixels, in which each pixel has two storage devices. During a single camera frame, for each pixel in the array, charge is transferred from the photodiode of the pixel at some times to one of the storage devices for the pixel, and at other times to the other storage device for the pixel. For example, during a single camera frame: (a) a modulation signal causes the then accumulated photodiode charge in pixel 1312 to be transferred to a capacitive bin at position 1311 at some times, and to be transferred to a capacitive bin at position 1303 at some other times; and (b) a separate modulation signal causes the then accumulated photodiode charge in pixel 1315 to be transferred to a capacitive bin at position 1314 at some times, and to be transferred to a capacitive bin at position 1316 at some other time.

[00265] After the end of the frame, the charges are shifted down each column of the array in a conventional CCD manner. For example, charge is read out from the capacitive bin at position 1316 at the edge of the array, then the charge in the capacitive bin at position 1315 is shifted to the capacitive bin at position 1316, then the charge in the capacitive bin at position 1314 is shifted to the capacitive bin at position 1315, then the charge in the capacitive bin at position 1313 is shifted to the capacitive bin at position 1314, then the charge in the capacitive bin at position 1312 is shifted to the capacitive bin at position 1313, then the charge in the capacitive bin at position 1311 is shifted to the capacitive bin at position 1312. Then the charge is read out again from the capacitive bin at position 1316 at the edge of the array, etc.

[00266] Figure 14 is a conceptual diagram of CCD pixels, in which each pixel has four storage devices. During a single camera frame, for each pixel in the array, charge is transferred from the photodiode of the pixel at some times to the first storage device for the pixel, at other times to the second storage device for the pixel, at yet other times to the third storage device for the pixel, and at yet other times to the fourth storage device for the pixel. For example, during a single camera frame, a modulation signal causes the then accumulated photodiode charge in pixel 1411 : (a) to be transferred to a capacitive bin at position 1453 at some times; (b) to be transferred to a capacitive bin at position 1455 at some other times; (c) to be transferred to a capacitive bin at position 1457 at some times; and (d) to be transferred to a capacitive bin at position 1459 at some other times. After the end of the frame, the charges are shifted down each column of the array in a conventional CCD manner.

[00267] In a pixel with two storages, one storage is by convention considered to be the primary storage. The "on" state of modulation of such a pixel means that the charge is transferred to the primary storage. The "off state of the modulation of such a pixel means that the charge is transferred to the secondary storage. The image recovered from the primary storages is the "positive" image, accumulated during the "on" phases. The image recovered from the secondary storages is the "negative" image, accumulated during the "off phases. Summing these two together gives the total image, i.e. an image which would be captured by an ordinary (unmodulated) camera.

[00268] As used herein, a "multiple storage implementation" or "total charge implementation" means an implementation in which each pixel has multiple storage devices that receive charge only from that pixel during the exposure time of a frame. Such an implementation is sometimes called a "total charge implementation" to emphasize that charge is not, by design, drained at some times during the frame without being measured.

[00269] Multiple storage implementations are advantageous for at least two reasons.

[00270] First, in some multiple storage implementations, a computer calculates a dynamic threshold depending on the total illumination for a particular pixel. For example, in some cases, instead of having a fixed threshold for deciding whether a pixel's value represents a logical 0 or logical 1 in code C _xy (see discussion of Figure 4), a computer compares the pixel's value to a threshold computed, for example, as 50% of the total value.

[00271] Second, in some multiple storage implementations, a computer calculates a total image for a single frame, by summing, for each pixel, the sum of the charges captured by the multiple storage devices for that pixel during that frame.

[00272] In most implementations, the modulation signals (such as signals 311-315 for superpixel modulation signal in Figure 3 and signals 3802 for curtain modulation in Figure 38 J) do not directly control timing within the pixel. Instead, electronic circuitry in the image sensor, or in the camera circuitry, or in the control unit converts each modulation signal to one or more intermediate signals, which directly or indirectly control timing within each pixel or pixel set, such as the timing of (i) photodiode reset, (ii) floating diffusion reset, (iii) readout, (iv) transfer of charge from photodiode to floating diffusion or to a capacitive bin; or (v) CCD shifts, in the case of a CCD. For example, in some implementations using CMOS pixels, the intermediate signals comprise high or low (binary values) of voltage applied to TX, TXi, TXil, TXi2, TXi3, RST, RST1, RST2, RST3, TD ROW, PD RST, PD RST ROW or CURTAIN ROW. In some cases, changes in an intermediate signal are triggered by the rising edge, falling edge or both of the modulation signal. In some cases, changes in an intermediate signal occur at a specified delay after the rising edge, falling edge or both of the modulation signal. In some cases, changes in an intermediate signal are the logical complement of the modulation signal (i.e., 0 if the modulation signal is 1, or 1 if the modulation signal is 0). In some cases, an intermediate signal that triggers transfer of charge from a photodiode to floating diffusion (i) becomes high when the modulation signal goes from high to low, stays there for a small amount of time necessary to transfer the charge, and goes low; and (ii) ignores the low-to-high transition of modulation signal.

[00273] Delayed signals are desirable in some cases when it is known that different elements of the pixel circuit have different latencies, or require different times to transfer the charge.

[00274] In some implementations, constraints imposed by the hardware

implementation require that the modulation signals satisfy one or more logical conditions. For example, in some multiple storage implementations, the following logical condition must be satisfied: at any one time, at most one modulation signal per pixel is in a state that triggers transfer of charge. This is because if more than one transfer gate is open, the charge from photodiode splits between the storages, which is undesirable. In some other

implementations, a logical condition is that storage modulation signals add up to 1.

[00275] This invention is not limited to binary coding. As used herein, "n-ary code" means code in which the digits are expressed in base n, where n is an integer greater than 2. For example, 3-ary code is written in base 3, and is sometimes called ternary code.

[00276] In some embodiments of this invention, ternary code is used with pixels that each have two storage devices. In that case, three values of the ternary code ("A","B","C") signal causes the charge to be transferred into storage "A", into storage "B", or trashed, respectively. In some cases: (a) each pixel has three storage devices; (b) ternary code is used; (c) instead of trashing the light in the "C" case, the charge is transferred into the third storage ("C"); and (d) A+B+C are summed to get the total amount of light.

[00277] In some implementations that employ n-ary code, each pixel has n-1 or n storages. With n-1 storage devices per pixel, the n-ary code is recoverable. With n storage devices per pixel, the n-ary code and total amount of light are recoverable.

[00278] In some implementations that employ n-ary code and that use n storage devices per pixel, the values 0, 1,..., n-1 of the n-ary code cause the charge to be transferred into the 1st, 2nd, ... , nth storage, respectively. The total illumination image is obtained by summing all values of the 1st, 2nd, ... , nth storage.

[00279] In some implementations that employ n-ary code and that use n-1 storage devices per pixel, the value 0 causes the photodiode charge to be drained, in some cases without being saved for later readout. The values l,...,n-l of the n-ary code cause the charge to be transferred into the 1st, 2nd, ... , (n - l)" ¹ storage, respectively. [00280] In some implementations, modulation of pixel storage is controlled by an n- ary code, such that all of the photodiode charge of each pixel is stored for later readout.

Figure 15 shows an example of this approach. In Figure 15, each pixel has n storage devices, and storage modulation for the /c ^tft pixel in a superpixel includes the following steps: A computer evaluates x, which is the value of the k ^th digit of an n-ary code of length m (Step 1501), and causes the photodiode charge in the /c ^tft pixel to be transferred to the

(x + l) ^tft storage device for the /c ^tft pixel. (Step 1503). For example, if x equals 0, then the pixel's photodiode charge is transferred to the first storage device for the pixel. For example, if x equals n— 1, then the pixel's photodiode charge is transferred to the n ^th storage device for the pixel. If a superpixel has more than one pixel that is a member of the k ^th pixel set and the method shown in Figure 15 is used, then storage modulation for each of these pixels is done simultaneously, in the manner shown in Figure 15.

[00281] In some implementations, modulation of pixel storage is controlled by an n- ary code, such that only a portion of the photodiode charge of each pixel is stored for later readout. Figure 16 shows an example of this approach. In Figure 16, each pixel has multiple (e.g., n-1) storage devices, and storage modulation for the /c ^tft pixel in a superpixel includes the following steps: A computer evaluates x, which is the value of the k ^th digit of a modulation code. (Step 1601) If x equals 0, then the photodiode charge in the k ^th pixe\ is drained, and is not saved for later readout. (Step 1603) If x is not equal to 0, then the photodiode charge in the /c ^tft pixel is transferred to the (x + l) ^tft storage device for the /c ^tft pixel. (Step 1605).

[00282] If a superpixel has more than one pixel that is a member of the k ^th pixel set and the method shown in Figure 16 is used, then storage modulation for each of these pixels is done simultaneously, in the manner shown in Figure 16.

[00283] Figure 17 is a flowchart for decoding n-ary code, after storage modulation in which only a portion of the photodiode charge is stored for later readout. In Figure 17, the code that is being decoded has m digits, and each pixel has multiple (e.g., n-1) storage devices. Decoding for each superpixel includes the following steps: A computer sets k equal to zero (Step 1701). A computer evaluates if k is greater than m-1. (Step 1703) If yes, then decoding stops for that superpixel, and the computer repeats the decoding for each of the superpixels, until all of the superpixels are decoded. (Step 1715) If no, then a computer detects the storage value (i.e., level of charge stored in a storage device) for each of the multiple storage devices of the k ^th pixel. (Step 1705) A computer determines if any of these storage values exceed a threshold. (Step 1707). If none of these storage values exceed the threshold, then a computer sets the k ^th digit of the code to zero. (Step 1709) If at least one of these storage values exceeds the threshold, then a computer: (a) determines the maximum storage value, out of these storage values (Step 1711), (b) determines that the maximum storage value is stored in the x ^th storage device for the k ^th pixel (Step 1712), (c) sets the k ^th digit of the code to x (Step 1713) and (d) sets k equal to k+1 (Step 1714)

[00284] If a superpixel has more than one pixel that is a member of the k ^th pixel set, then the method in Figure 17 is modified for that superpixel so that: (a) in Step 1705, a computer determines storage values for each of the storage devices for each of the pixels in the superpixel that are members of the k ^th pixel set; and (b) in Step 1707, a computer determines that the maximum storage value, out of these storage values, is stored in the x ^th storage device of one of these pixels.

[00285] Figure 18 is a flowchart for decoding n-ary code, after storage modulation in which all of the photodiode charge is stored for later readout. In Figure 18, the code that is being decoded has m digits, and each pixel has n storage devices. Decoding for each superpixel includes the following steps: A computer sets k equal to zero (Step 1801). A computer evaluates if k is greater than m-1. (Step 1803) If yes, then decoding stops for that superpixel, and the computer repeats the decoding for each of the superpixels, until all of the superpixels are decoded. (Step 1815) If no, then a computer detects the storage value (i.e., level of charge stored in a storage device) for each of the multiple storage devices of the k ^th pixel. (Step 1805) A computer: (a) determines the maximum storage value, out of these storage values (Step 1807), (b) determines that the maximum storage value is stored in the x ^th storage device for the k ^th pixel (Step 1809), (c) sets the k ^th digit of the code to x-1 (Step 1811), and sets k equal to k+ 1 (Step 1813).

[00286] If a superpixel has more than one pixel that is a member of the k ^th pixel set, then the method in Figure 17 is modified for that superpixel so that: (a) in Step 1705, a computer determines storage values for each of the storage devices for each of the pixels in the superpixel that are members of the k ^th pixel set; and (b) in Step 1707, a computer determines that the maximum storage value, out of these storage values, is stored in the x ^th storage device of one of these pixels.

[00287] In some implementations in which each pixel stores charge to only one storage device: (a) a binary code is applied to the pixels; and (b) each pixel is turned "off and "on" many times during the duration of a single camera frame. [00288] In many implementations, a pixel is neither "on" during an entire frame, nor "off during an entire frame. (Putting a pixel with only a single storage device to constant "off during an entire frame would cause the pixel not to measure any light during the frame. Putting a pixel with only a single storage bin to constant "on" for the whole frame duration would cause the pixel to measure light normally during the entire frame.)

Analog Modulation

[00289] In some cases, the modulation signals - for superpixel modulation, curtain modulation, or both - are analog signals. In some cases, modulations signals (either digital or analog) undergo at least one change between camera frames. Figure 19 is diagram that illustrates three examples of analog modulation signals that change from frame to frame. Signal 1901 is an aperiodic signal in both frame 1 and frame 2. Signal 1903 is a periodic signal that is sinusoidal in frame 1 and non-sinusoidal in frame 2. In frame 1, signal 1905 is a periodic triangle wave that is shifted upward so that the minimum value is zero. In frame 2, signal 1905 is aperiodic.

[00290] In some cases, an analog modulation signal is negative at times, such as signal 1901 at time 1907 in Figure 19. In some cases, a "negative" value of a modulation signal is implemented as follows: the storage is pre-charged by a fixed amount of charge. For a negative portion of the modulation signal, a transfer gate is opened to release an amount of charge from the storage device proportional to the absolute value of the modulation signal. During the readout phase, a computer treats the pre-charged amount as the zero value, treats an amount of charge below this value as a negative readout; and treats an amount of charge larger than the pre-charge amount as a positive readout.

[00291] For analog modulation, the pixel's sensitivity to light is not a binary state; instead the pixel may have a continuous range of sensitivity from completely "off to completely "on". This sensitivity may vary over time during each camera frame.

Light Modulation

[00292] In some implementations, modulation signals control the intensity of light incident on the photodiode of a pixel, for superpixel modulation, curtain modulation, or both. In these cases: (a) the amount of light that reaches the pixels is controlled by selective attenuation of reflected or transmitted light; and (b) thus, the sensitivity of pixels to light from the scene is controlled by controlling whether the light reaches the pixels. [00293] Figure 20A shows a reflective SLM 2003 that is controlled by the logical AND of the curtain modulation and superpixel modulation. In Figure 20A, the effect of this logical AND is that SLM 2003 reflects light from the scene such that, at any given time: (a) for each given pixel in the active region of the image sensor, light from the scene is reflected by SLM 2003 if the superpixel modulation for the given pixel is high; and (b) light from the scene is not reflected by SLM 2003 to the inactive region of the image sensor.

[00294] Figure 20B shows two reflective SLMs: the first SLM 2002 performs curtain modulation and the second SLM 2003 performs superpixel modulation (or vice versa). In Figure 20B, in order for light from the scene to reach pixels (e.g., 2007, 2009, 2011), the light must reflect from both SLM 2002 and SLM 2003 before reaching the pixels. Thus, in Figure 20B, the logical AND of the curtain and superpixel modulation signals is effectively taken. In Figure 20B, the effect of this logical AND is (similar to Figure 20A) that the two SLMs 2002, 2003, taken together, reflect light from the scene such that, at any given time: (a) for each given pixel in the active region of the image sensor, light from the scene is reflected by the SLMs if the superpixel modulation of the given pixel is high; and (b) light from the scene is not reflected by the SLMs to the inactive region of the image sensor. However, in Figure 20B (unlike Figure 20A), the modulation signals are performed separately: SLM 2002 performs curtain modulation and SLM 2003 performs superpixel modulation (or vice versa).

[00295] In the examples shown in Figures 20 A and 20B, light from a scene 2001 strikes reflective SLM(s) and is selectively reflected. Thus, which pixels in an image sensor are illuminated by reflected light from the SLM(s) at any given time during a camera frame is controllable. For example, which pixels (out of pixels 2007, 2009, 2011 in superpixel 2005), are illuminated by reflected light from the SLM(s) varies during a single camera frame. For example, in some cases, during a single camera frame: (a) at some times pixel 2007 is illuminated by reflected light from the SLM(s) but pixels 2009 and 2011 are not; and (b) at other times pixels 2007 and 2009 are illuminated by reflected light from the SLM(s) but pixel 2011 is not. In some cases, each reflective SLM comprises a DMD (digital micromirror device) or a reflective LCOS (liquid crystal on silicon) device. In some cases, each reflective SLM may comprise a planar reflective array, or a reflective array positioned on a curved surface (such as a surface of a cylinder or torus).

[00296] In Figure 21 A, a single reflective DMD array is controlled by the logical AND of the curtain and superpixel modulation signals. In Figures 2 IB and 21C, the two functions are performed by two DMD arrays: one DMD performs curtain modulation and the other DMD simultaneously performs superpixel modulation. In Figure 2 IB, light reflects from the DMD that performs superpixel modulation, and then from the DMD that performs curtain modulation. In Figure 21C, the order is reversed. In Figures 2 IB and 21C, the effect of the two simultaneous modulations is that the light reflected from the two DMDs (taken together) to the image sensor is controlled by the logical AND of the curtain and superpixel modulation signals.

[00297] Figures 21 A, 21B, and 21C show DMD array(s) selectively reflecting light from a scene at a particular time, such that reflected light is steered toward some, but not all, of the pixels of an image sensor. In the examples shown in Figures 21 A, 21B, and 21C, the pixels in the image sensor and pixels of the DMD array(s) are optically aligned, such that each pixel of a DMD corresponds to a pixel of the image sensor. The pixels in the DMD array(s) are switched between two states, "A" and "B". When the mirror for a DMD pixel is in the A state, the mirror reflects the light in an optical path (directly or indirectly) toward the corresponding pixel in the image sensor. When the mirror for a DMD pixel is in the B state, the light is reflected in another direction. By switching a DMD pixel between these two states, the light incident on the corresponding pixel of the image sensor may be modulated, and thus the response of the pixel to the light coming from the scene may be modulated.

[00298] Figure 21 A is a ray diagram that shows a DMD that is controlled by the logical AND of the curtain and superpixel modulation signals, and thus may be loosely said to perform both current modulation and superpixel modulation simultaneously. The DMD selectively reflects light from a scene. In Figure 21 A, pixels in a DMD array 2105 are optically aligned with pixels in image sensor 2103. For example, DMD pixels 2111, 2113, 2115 correspond to image sensor pixels 2121, 2123, 2125 respectively. Light from a scene 2101 is striking DMD pixels 2111, 2113 and 2115. DMD pixels 2111 and 2113 are in state "A", such that light reflects from these pixels to image sensors 2123 and 2123, respectively. DMD pixel 2115 is in state "B", such that light reflects from this DMD pixel in a direction away from image sensor 2125, thus causing the reflected light not to reach the image sensor.

[00299] Figure 2 IB is a ray diagram that shows light 2101 from a scene traveling to a DMD array 2105 that performs superpixel modulation, then to a DMD array 2130 that performs curtain modulation, and then to an image sensor 2103. In Figure 2 IB, pixels in image sensor 2103 are optically aligned with pixels in DMD array 2130 and with pixels in DMD array 2105. For example, pixels 2121, 2123, 2125 in image sensor 2103 correspond to pixels 2141, 2143, 2145, respectively, in DMD array 2130 and to pixels 2111, 2113, 2115, respectively, in DMD array 2105. In Figure 2 IB, DMD 2105 performs superpixel modulation. Light from a scene 2101 is striking (among other things) pixels 2111, 2113, 2115 in DMD 2105. Pixels 2111 and 2113 in DMD 2105 are in state A. Thus, light reflects from these pixels 2111, 2113 in DMD 2105 to pixels 2141, 2143, respectively, in DMD 2130. Pixel 2115 in DMD 2105 is in state B, and thus reflects light in a direction away from DMD 2130 and image sensor 2103. In Figure 21B, DMD 2130 performs curtain modulation. Pixel 2143 in DMD 2130 is in state A. Thus, light reflects from pixel 2143 to image sensor pixel 2123. Pixel 2141 in DMD 2130 is in state B. Thus, light reflects from pixel 2141 in a direction away from the image sensor 2103, thus causing the reflected light not to reach the image sensor.

[00300] Figure 21C is a ray diagram that shows light from a scene traveling to a DMD that performs curtain modulation, then to a DMD that performs superpixel modulation, and then to an image sensor. In Figure 21C, pixels in image sensor 2103 are optically aligned with pixels in DMD array 2105 and with pixels in DMD array 2130. For example, pixels 2121, 2123, 2124, 2125 in image sensor 2103 correspond to pixels 2111, 2113, 2114, 2115, respectively, in DMD array 2105 and to pixels 2141, 2143, 2144, 2145, respectively, in DMD array 2130. In Figure 21C, DMD 2130 performs curtain modulation. Light from a scene 2101 is striking (among other things) pixels 2141, 2143, 2144, 2145 in DMD 2130. Pixels 2143 and 2144 in DMD 2130 are in state A. Thus, light reflects from these pixels 2143, 2144 in DMD 2130 to pixels 2113, 2114, respectively, in DMD 2105. Pixels 2141, 2145 in DMD 2130 are in state B, and thus reflect light in directions away from DMD 2105 and image sensor 2103. In Figure 21C, DMD 2105 performs superpixel modulation. Pixel 2113 in DMD 2105 is in state A. Thus, light reflects from pixel 2113 in DMD 2105 to image sensor pixel 2123. Pixel 2114 in DMD 2105 is in state B. Thus, light reflects from pixel 2114 in a direction away from the image sensor 2103, thus causing the reflected light not to reach the image sensor.

[00301] In Figures 21B and 21C, dark shading in a DMD pixel indicates that the pixel is in the B state and thus reflects light away from the image sensor and the other DMD, if any.

[00302] Figure 22A shows a transmissive SLM 2203 that is controlled by the logical AND of the curtain modulation and superpixel modulation. In Figure 22A, the effect of this logical AND is that SLM 2203 transmits light from the scene such that, at any given time: (a) for each given pixel in the active region of the image sensor, light from the scene is transmitted through SLM 2203 if the superpixel modulation for the given pixel is high; and (b) light from the scene is not transmitted through SLM 2203 to the inactive region of the image sensor.

[00303] Figure 22B shows two transmissive SLMs: the first SLM 2202 performs curtain modulation and the second SLM 2203 performs superpixel modulation (or vice versa). In Figure 22B, in order for light from the scene to reach pixels (e.g., 2207, 2209, 2211), the light must be transmitted through both SLM 2202 and SLM 2203 before reaching the pixels. Thus, in Figure 22B, the logical AND of the curtain and superpixel modulation signals is effectively taken. In Figure 22B, the effect of this logical AND is (similar to Figure 22 A) that the two SLMs 2202, 2203, taken together, transmit light from the scene such that, at any given time: (a) for each given pixel in the active region of the image sensor, light from the scene is transmitted by the SLMs if the superpixel modulation for the given pixel is high; and (b) light from the scene is not transmitted by the SLMs to the inactive region of the image sensor. However, in Figure 22B (unlike Figure 22A), the modulation signals are performed separately: SLM 2202 performs curtain modulation and SLM 2203 performs superpixel modulation (or vice versa).

[00304] In the example shown in Figures 22 A and 22B, light from a scene 2201 passes through transmissive SLM(s), and is selectively attenuated. Thus, which pixels are illuminated by light passing through the SLM(s) at any given time during a camera frame is controllable. For example, which pixels (out of pixels 2207, 2209, 2211 in superpixel 2205), are illuminated by light passing through the SLM(s) varies during a single camera frame. For example, in some cases, during a single camera frame: (a) at some times pixel 2207 is illuminated by light passing through the SLM(s) but pixels 2209 and 2211 are not; and (b) at other times pixels 2207 and 2209 are illuminated by light passing through the SLM(s) but pixel 2011 is not. In some cases, the transmissive SLM comprises an LCD (liquid crystal display) or an array of ferromagnetic shutters. In some cases, each transmissive SLM may comprise a planar array, or an array positioned on a curved surface (such as a surface of a cylinder or torus).

[00305] In Figure 23 A, a single LCD (liquid crystal display) array is controlled by the logical AND of the curtain and superpixel modulation signals. In Figure 23B, the two functions are performed by two LCD arrays: one LCD performs curtain modulation and the other LCD simultaneously performs superpixel modulation. In Figure 23B, light is transmitted by the LCD that performs curtain modulation, and then by the LCD that performs superpixel modulation. In Figure 23B, the effect of the two simultaneous modulations is that the light transmitted through the two LCDs (taken together) to the image sensor is controlled by the logical AND of the curtain and superpixel modulation signals.

[00306] Figures 23 A and 23B show LCD(s) selectively attenuating light from a scene at a particular time, such that light passing through the LCD(s) reaches some, but not all, of the pixels of a superpixel in an image sensor. In the examples shown in Figure 23 A and 23B, the pixels in the image sensor and pixels of the LCD(s) are optically aligned, such that each pixel of an LCD corresponds to a pixel of the image sensor. The pixels in the LCD array (s) are switched between two states, "on" and "off. When an LCD pixel is in the "on" state, light from the scene passes through the LCD pixel. When an LCD pixel is in the "off state, the LCD pixel is opaque and blocks light. By switching an LCD pixel between these two states, the light incident on the corresponding pixel of the image sensor may be modulated, and thus the response of the pixel to the light coming from the scene may be modulated.

[00307] Figure 23 A is a ray diagram that shows an LCD that is controlled by the logical AND of the curtain and superpixel modulation signals, and thus may be loosely said to perform both current modulation and superpixel modulation simultaneously. The LCD selectively attenuates light from a scene. In Figure 23 A, pixels in LCD 2303 are optically aligned with pixels in image sensor 2309. For example, LCD pixels 2311, 2313, 2315 and 2317 correspond to image sensor pixels 2321, 2323, 2325 and 2327, respectively. Light from a scene 2301 is striking LCD pixel s 2311 , 2313, 2315 and 2317. LCD pixel s 2311 , 2313 , 2315 are in the "on" state, such that light passes through these LCD pixels and travels to image sensors 2321, 2323 and 2325, respectively. LCD pixel 2317 is in the "off state, such that LCD pixel 2317 is opaque, thus preventing light from passing through LCD pixel 2317 and traveling to image sensor pixel 2327. Image sensor pixels 2321, 2323, 2325 and 2327 are all in the same superpixel.

[00308] Figure 23B is a ray diagram that shows light from a scene traveling to an LCD that performs curtain modulation, then to a LCD that performs superpixel modulation, and then to an image sensor. In Figure 23B, pixels in image sensor 2309 are optically aligned with pixels in LCD 2303 and with pixels in LCD 2330. For example, pixels 2321, 2323, 2327 in image sensor 2309 correspond to pixels 2311, 2313, 2317, respectively, in LCD 2303 and to pixels 2331, 2333, 2337, respectively, in LCD 2330. In Figure 23B, LCD 2330 performs curtain modulation. Light from a scene 2301 is striking (among other things) pixels 2331, 2333, 2334, 2336, 2337, 2338 of LCD 2330. Pixels 2331, 2333, 2337 in LCD 2330 are in the "on" state, such that light passes through these LCD pixels and travels to pixels 2311, 2313, 2317, respectively, in LCD 2303. Pixels 2334, 2336, 2338 in LCD 2330 are in the "off state, and thus are opaque and prevent light from passing through them. In Figure 23B, LCD 2303 performs superpixel modulation. Pixels 2311, 2313 in LCD 2303 are in the "on" state, such that light passes through these LCD pixels and travels to pixels 2321, 2323, respectively, in image sensor 2309. LCD pixel 2317 is in the "off state, such that LCD pixel 2317 is opaque, thus preventing light from passing through LCD pixel 2317 and traveling to image sensor pixel 2327.

[00309] In Figures 23 A and 23B: (a) dark shading in an LCD pixel indicates that the pixel is in the "off state and thus is opaque and blocks light; and (b) dark shading in an image sensor pixel indicates that the pixel does not receive light from the scene because light from the scene is blocked by LCD(s).

[00310] In Figures 23A and 23B, a camera 2307 includes a lens 2305 and the image sensor 2309. The lens bends light from LCD 2303. Similarly, in some cases: (a) a lens is interposed between a reflective SLM (such as a DMD or reflective LCOS) and the image sensor, and (b) the lens bends light rays that travel from the reflective SLM through the lens to the reflective SLM.

[00311] Figures 24A-24H show different configurations of an optical system that includes transmissive SLM(s), in illustrative implementations of this invention. In Figures 24A-24H, the SLM(s) selectively attenuate light, such that at different times during a single camera frame, the SLM(s) block light from reaching different pixels in the image sensor.

[00312] In Figures 24A, 24B and 24C, the transmissive SLM is controlled by the logical AND of the superpixel modulation and the curtain modulation, and thus may be loosely said to perform both of these modulations simultaneously.

[00313] In Figure 24A, light from a scene 2403 travels through an imaging lens 2407, then through a transmissive SLM 2401, and then to an image sensor 2405.

[00314] In Figure 24B, light from a scene 2403 travels through a transmissive SLM

2401, then through an imaging lens 2407, and then to an image sensor 2405.

[00315] In Figure 24C, light from a scene 2403 travels through an imaging lens 2407, then through a transmissive SLM 2401, then through a field lens 2411, then through a relay lens 2409, and then to an image sensor 2405.

[00316] Figures 24D, 24E, 24F, 24G and 24H are each a diagram of an optical system that includes two transmissive SLMs: specifically, an SLM that performs superpixel modulation and an SLM that performs curtain modulation. In Figures 24D-24H, the logical AND of the curtain and superpixel modulation signals is effectively taken. In Figures 24D- 24H, the effect of this logical AND is (similar to Figure 20A) that two transmissive SLMs (e.g., 2421, 2425), taken together, transmit light from the scene such that, at any given time: (a) for each given pixel in the active region of the image sensor, light from the scene is transmitted through the SLMs if the superpixel modulation for the given pixel is high; and (b) light from the scene is not transmitted through the SLMs to the inactive region of the image sensor.

[00317] In Figure 24D, light from a scene 2403 travels through an imaging lens 2407, then through a transmissive SLM 2425 that performs curtain modulation, then through a transmissive SLM 2421 that performs superpixel modulation, and then to an image sensor 2405.

[00318] In Figure 24E, light from a scene 2403 travels through an imaging lens 2407, then through a transmissive SLM 2421 that performs superpixel modulation, then through a transmissive SLM 2425 that performs curtain modulation, and then to an image sensor 2405.

[00319] In Figure 24F, light from a scene travels through a transmissive SLM 2425 that performs curtain modulation, then through a transmissive SLM 2421 that performs superpixel modulation, then through an imaging lens 2407, and then to an image sensor 2405.

[00320] In Figure 24G, light from a scene travels through a transmissive SLM 2421 that performs superpixel modulation, then through a transmissive SLM 2425 that performs curtain modulation, then through an imaging lens 2407, and then to an image sensor 2405.

[00321] In Figure 24H, the optical system also includes a relay lens 2409. Figure 24H shows positions in which a transmissive SLM (that performs curtain modulation, superpixel modulation, or that is controlled by the logical AND of the two) may be located in the optical system. Specifically, in the example shown in Figure 24H, such a transmissive SLM (that performs curtain modulation, superpixel modulation, or that is controlled by the logical AND of the two) may be positioned at any of locations 2431, 2432, 2433, 2435 or 2437. For example: (a) a transmissive SLM that is controlled by the logical AND of the curtain and superpixel modulations may be positioned at any of those locations; or (b) a transmissive SLM that performs curtain modulation may be positioned at one of these locations and another transmissive SLM that performs superpixel modulation may be positioned at another of those locations. [00322] In some implementations in which an SLM is used, a computer separately controls each pixel in the SLM, and thus the computer controls the size and shape of a superpixel in the image sensor. For example, in some cases, a computer sends signals to a reflective or transmissive SLM, such that 9 pixels comprise a 3x3 pixel superpixel, or such that 12 pixels comprise a 4x3 pixel superpixel, or such that 16 pixels comprise a 4x4 pixel superpixel.

[00323] In some cases, a reflective or transmissive SLM does not have the same size or shape as the image sensor. In those cases: (a) some pixels in the image sensor do not correspond to pixels in the SLM; and (b) light modulation is applied only to pixels in the image sensor that are aligned with pixels in the SLM.

[00324] In some cases: (a) the pixels in the SLM and camera are not precisely aligned; and (b) pixels for which the imprecision exceeds a given threshold are ignored. The value at that position is interpolated or returned as "unknown".

[00325] In many cases where a reflective or transmissive SLM is used, each pixel has only storage device that is dedicated to that pixel and does not receive charge from any other pixel during the exposure time of a camera frame.

Comparison to Time-Multiplexed Structured Illumination

[00326] It is interesting to compare this invention to conventional time-multiplexed structured illumination.

[00327] Figure 25 is a conceptual diagram that shows a prior art example of conventional time-multiplexed structured illumination. In Figure 25, the camera 2511 sensitivity is not modulated. The laser 2503 and a Powell lens (not shown) project a plane of laser light 2505. The line of laser light 2505 sweeps across the scene during each camera frame. The intensity of the laser light is modulated by a binary modulation pattern, causing the laser light to turn on and off repeatedly during a single sweep. The camera captures a single frame during the sweep of the laser light across the scene. In the image 2517 captured during this frame, the dark stripes 2533, 2535, 2537, 2539 correspond to the times during the sweep at which the laser light is turned off.

[00328] In conventional time-multiplexed structured illumination (such as that shown in Figure 25): (a) n separate modulation signals that comprise the n-bit code are applied during n separate frames of the camera, one modulation signal per frame, to modify illumination of the scene; and (b) the sensitivity of the camera to light is not modulated. [00329] In contrast, in illustrative implementations of this invention, all n separate modulation signals that comprise an /7-bit code are applied during a single frame of a camera, to separately modify the response of n pixels in each superpixel to incident light.

[00330] In illustrative implementations, this invention captures (albeit at a lower spatial resolution) in a single camera frame the same type of data as is captured during multiple camera frames with time-multiplexed structured illumination. Capturing all of the data in a single frame (in illustrative implementations of the present invention) rather than in multiple frames (as in conventional time-multiplexed structured illumination) has the desirable effect of reducing distortion and artifacts in 3D sensing that are caused by motion blur.

3D Scanning

[00331] In illustrative implementations, this invention is used for 3D scanning - i.e., for detection of the 3D position of points in a scene.

[00332] In illustrative implementations, a system comprises at least one camera with a modulatable image sensor, an optional ordinary camera, a control unit, a processing unit, and at least one illumination unit. Modulation of pixels of a superpixel during a camera frame creates a specific code in the pixels of the superpixel

[00333] In this system, a light source emits light, such that the position of the intersection of the light and the scene changes over time.

[00334] In illustrative implementations, the spatio-temporal characteristics of the light source are such that the spatial position of a point of the scene captured by a superpixel is computationally recoverable from the camera calibration, the relative position of the light source and the camera, and the charge values recorded by the pixels of the superpixel during a single frame of a camera.

[00335] In illustrative implementations, the spatiotemporal characteristics of the light source are such that: (a) for a given point in the scene, the time function of the intensity of direct illumination from the light source in a point of the scene exhibits only one peak; and (b) the intersection of (1) the set of the points illuminated at time t and (2) the pixel ray of the superpixel has only one solution, given constraints consisting of the field of view of the camera, light source field of view, angle of the light emitted by the light source, and the relative position of camera and the light source. [00336] Figure 26 is a conceptual diagram that illustrates a moving dot of light produced by controlled movement of a light source (not shown in Figure 26). For example, in some cases the light source comprises an object that emits light, such as a laser or directional LED. In other cases, the light source comprises a specular surface (such as a galvanometer-actuated mirror) that reflects light. In the example shown in Figure 26, a dot of light moves across a scene, such that the dot of light intersects the scene at different spatial positions 2601, 2603, 2605, 2607, 2609 at different times.

[00337] Figures 27A, 27B and 27C show three examples of light sources that produce a moving beam of light, in illustrative implementations of this invention. For example, the light beam may move in the same path in each camera frame. At different times of each camera frame, the path may traverse all parts of a scene (e.g., by moving the light beam in repeated vertical movements, from bottom to top of the scene, with a small horizontal displacement after each vertical movement).

[00338] In Figures 27 A, 27B and 2C, the moving light beam illuminates the scene with a moving dot of light. The moving dot of light may be used for curtain modulation (in order to suppress ambient light) and for 3D scanning.

[00339] In illustrative implementations, the moving light beam is oriented, at different times in each camera frame, in different directions that are not co-planar with each other (i.e., do not all lie in the same geometric plane). Alternatively, in some cases, the light beam may point in directions that lie in only one geometric plane.

[00340] Figure 27A is a diagram, which shows a rotating mirror that reflects a moving laser beam. The mirror 2701 rotates about axis of rotation 2703, which axis is perpendicular to Figure 27A. A laser 2705 shines a laser beam. As the mirror rotates, the angle of the laser beam varies as a function of time. For example, at one time, the laser beam may be pointed in direction 2711, and at another time the laser beam may be pointed in direction 2713.

[00341] Figure 27B is a diagram, which shows a projector 2721 that projects a moving light beam. The light beam is pointed in different directions at different times. For example, the light beam is pointed, at different times, in directions 2723, 2725, 2727.

[00342] Figure 27C is a diagram, which shows an array of directional light sources (e.g., 2731, 2733, 2735, 2737) that, taken together, emit a moving light beam. The light beam is pointed in different directions at different times. For example, the light beam is pointed, at different times, in directions 2741, 2743, 2745. [00343] In illustrative implementations, 3D scanning is implemented as follows, for each superpixel: (a) n modulation signals are applied to n separate sets of pixels in a superpixel during a single camera frame, one modulation signal per pixel set, such that at least pixel in the superpixel is a member of each pixel set; (b) each modulation signal modulates the response of pixels to light from a scene , which light is incident on the pixel unit; (c) a computer determines a unique permutation of the n modulation signals, which permutation occurred at a time (the "hit time") when the intensity of light reflected from the scene and incident on the pixel unit reached a maximum for the frame; (d) a computer uses this unique permutation to perform a first algorithm to compute 3D coordinates of a point in the scene, which point is the position in the scene from which light reflected directly to the superpixel at the hit time; and (e) the first algorithm includes computing the 3D coordinates of the point by triangulation or includes accessing values from a lookup table, which lookup table was computed by a second algorithm that includes computing 3D depth by

triangulation.

[00344] In some implementations, depth-sensing is implemented as follows: The n pixels in the superpixel record n separate modulation signals during a single frame of the camera, which correspond to the n bits of an /7-bit code. Two rotating mirrors cause a dot of laser light to move in the same path through a scene during each camera frame. There are 2 ⁿ permutations of the n-bit code, corresponding to 2 ⁿ timeslots in each camera frame. The intensity of incident light at the superpixel is highest during a single timeslot in which the laser light reflects from the scene to the superpixel. Based on the permutation of the n modulation signals that is recorded by the superpixel at the "hit point" (when the intensity of incident light is highest), a computer determines the timeslot during which the hitpoint occurred. This timeslot in turn uniquely identifies the orientation of the beam of laser light when the hitpoint occurred. From the orientation of the laser beam, and from the known positions of the light source and of the pixel ray for the pixel, a computer calculates the 3D coordinates of the intersection of (i) the pixel ray and (ii) a vertical plane that is parallel to and that intersects the laser beam. This intersection (of the pixel ray and the vertical plane) is located at the scene point that directly reflected the laser light to the superpixel. This process is repeated for each scene point, yielding a cloud of computed 3D points in the scene.

[00345] In some 3D scanning implementations of this invention, one or more processors (e.g., in processing unit 2805 in Figure 28) perform calculations that take the camera output, calibration, and other static or dynamic parameters, and produce a description of the scene, including a 3D reconstruction of the surface geometry of the scene.

[00346] In some 3D scanning implementations of this invention, binary Gray code of length 9 is used for pixel set modulation. The illumination subsystem projects a sweeping laser line. The 9-bit vector represents the time t (within the frame 0 < t < Tf _rame ) at which the laser dot passed that part of the image which contributes to the camera pixel illumination (under the assumption that the scene is consistent - i.e., that there is no excessive reflection and scattering). The output of the camera pixels is used to decode the 9-bit vector and the time t within the frame. The time t identifies the orientation of the laser beam and therefore identifies a vertical plane that is parallel to, and that intersects, the laser beam. Based on this information - and the fact that the imaging system is aligned with known parameters - one or more processors identify a point in 3D space using a triangulation technique.

[00347] Figure 28 is a block diagram of hardware components of a 3D scanning system, which components comprise an illustrative implementation of this invention. The scanning system comprises an illumination subsystem 2801, control unit 2803, processing unit 2805, and camera subsystem 2807. In this system, one or more signal generators (e.g., 2802, 2820, 2806) or one or more computers (e.g., control unit 2803 or processing unit 2805) causes separate modulation signals to be created, which modulation signals control the sensitivity of multiple sets of pixels in the image sensor 2809 in camera subsystem 2807.

[00348] The signal generators (e.g., 2802, 2820, 2806) comprise one or more signal generators, function generators, digital pattern generators, frequency generators or any other electronic device that generates repeating or non-repeating electrical signals, including either digital or analog signals. The system includes one or more computers (e.g., control unit 2803 or processing unit 2805), which include memory devices (e.g., 2804, 2810)

[00349] The illumination subsystem 2801 may comprise a laser, two galvo mirror and optionally a lens.

[00350] The camera subsystem 2807 comprises optical elements 2808 and image sensor 2809. The optical elements 2808 comprise any optical elements used to transfer or direct light from the scene to the sensor, or to filter the light from the scene. The optical elements include lenses 2820 (e.g., adjustable lenses) arranged in an appropriate (static or dynamic) position, and other optical elements 2822. For example, in some cases, the other optical elements 2822 include (i) one or more spatial light modulators, mirrors, filters, optical fibers, or other light modulators or devices for attenuating, transferring, filtering, or directing light, and (ii) any adjustable optical elements that transfer, or direct light from the scene to the sensor (e.g., adjustable aperture shape, adjustable aperture size, adjustable aperture position, or other), or to filter the light from the scene (e.g. wavelength, polarization, incident angle, incident position of other). The adjustable optical elements are controlled by control unit 2803.

[00351] The camera subsystem 2807 is a system for capturing images of the scene 2811 using image sensor 2809. The image sensor consists of plurality of pixels (e.g., thousands of pixels, or millions of pixels, or billions of pixel, or any other number).

[00352] During 3D scanning, the galvanometer-actuated optical scanners control the position of mirrors and thus the direction of the projected dot of laser light.

[00353] In many cases, the position/orientation of camera subsystem and illumination subsystem is fixed for the life-time of the system. The illumination subsystem comprises the controlled laser light deflection system. The baseline is the distance between camera center and the light deflection system (the position of galvanometer-actuated mirror closest to the scene). The control unit 2803 controls the orientation (angle) of the galvanometer-actuated mirrors by applying time-changing electrical signals. The set of rotating mirrors deflect the laser beam in different directions, causing a dot of laser light to move to different positions in a scene. The value of the control signal is continuous and is any signal between minimal value (correspond to boundary angle of the mirror - _m m) and maximal value (correspond to second boundary angle of the mirror - aw). The angle of each mirror changes from _m m to Ctrnax in linear fashion proportional to control signal.

[00354] The operator calibrates the system at least once. During the calibration, the operator uses an automatic/semi-automatic calibration process to calibrate internal parameters of the camera.

[00355] In illustrative implementations, the optical system that is being calibrated includes two rotating mirrors, with axes of rotation that are perpendicular to each other, as shown in Figure 29B. These two mirrors, by each rotating by different amounts, deflect the laser beam in different directions. To calibrate the mutual position/orientation of the camera subsystem and the laser beam for all angles of the first mirror while the second mirror is at a particular angle ?, an operator performs triangulation. The second mirror is held steady at a particular angle ?, while the first mirror is rotated through its entire range of angle . While the first mirror is rotated through its entire range of angle a (and the second mirror is held constant at the particular angle ?), the position/orientation is calibrated in predefined angles m (i G 0 ... n-1; and an-i = a ). The algorithm calculates, for every angle of deflection mirror m, the equation of a vertical plane that is parallel to (and intersects) the laser beam, as follows: m _x * x + m _y * y + m _z * z + di = 0, where x, y, z are Euclidian coordinates in world coordinate space; ni is the 3D normal vector of the plane (the angle between ni and ni+i < π rad). And di is the orthogonal distance between the plane and the center of the origin.

[00356] The computed equations are stored in memory. In runtime, the algorithm computes the equation of the plane for any other between m and m+i (m < m+i) using interpolation. While the angle of the mirror is controlled by linear signal, a computer calculates the exact plane equation for any a\ (the algorithm computes new normal vectors n and d. At first, it uses binary search to find indexes i and i+1. The n lies in same plane as ni and ni+i. The angle 4-1 n ; nj : 4-1 n ; m+i l = I a - and 4-1 n ; ni+i l + 4-1 n ; ni+i l = 4-1 ni ; ni+i l (symbol "4-1 ; l " means the angle between vectors, symbol ":" means the ratio between two real values). The distance d = di + [( - m) I (m+i - m)] * (di+i - di) - (symbol "/" means division of real numbers and symbol "*" means multiplication of real numbers).

[00357] The calibration procedure described in the previous two paragraphs is then repeated for different angles ?, until calibration has been performed for all combinations of angle of the first mirror and angle β of the second mirror. For each angle β of the second mirror, the entire range of angle of the first mirror is calibrated, as described above.

[00358] During operation, the control unit controls the angle of the deflection mirrors and the modulation signals to individual sets of pixels. The system works in cycles (frames). One cycle comprises a setup phase, exposure phase, readout phase, processing phase and output phase. The individual phases are processed in a pipeline, so that the exposure phase of the current frame is executed simultaneously with the processing phase of previous frame.

[00359] During the setup phase, the control unit prepares the illumination subsystem and the camera subsystem for the upcoming exposure phase. This preparation includes image sensor reset, preparation of modulation signals (from the modulation functions), setting of image sensor exposure, setting the angle of deflection mirror to aw, and adjusting the power of the laser. In some implementations, control unit perform other preparatory steps.

[00360] During the exposure phase, the control unit sends the signal to illumination subsystem, to turn on the laser source and to change the angle of deflection mirrors. If T is the time of frame exposure in seconds (0 < t < T s). Control unit changes the angle for each mirror, such that = tfmin + (t / T) * (aw - aw).

[00361] In some implementations, the dependency between angle of deflection and the time during exposure is nonlinear.

[00362] The control unit applies the modulation signals to individual pixel sets. For example, in some cases, a 9-bit binary Gray code is used to drive the modulation of individual pixel sets. The 9-bit Gray code can encodes 512 discrete codewords Cxy. The time T is divided into 512 individual time periods Ti = <ti, ti+i) (i G 0 ... 511) of the same duration (|Ti| = T / 512), so that, in succession of the coding, each time period have its own 9 bit codeword (000000000, 000000001, 000000011, 000000010, ....). During the frame, in any time t (thit) fits into some interval Ti. During the interval Ti, the associate codeword is used by control unit to modulate response of the pixel sets. The first pixel set is controlled by first bit of the codeword, the second set is controlled by second bit of the codeword, ... , the ninth set is controlled by ninth bit of the associated codeword (if particular bit is 0, the all pixels from the particular pixel set are "off, if the bit is 1, the pixels are "on" - responsive to light).

[00363] The method described here for a 9-bit gray code is the same as the method for a 5-bit gray code that is shown in Figure 3, with appropriate modifications for the different number of bits in the code.

[00364] During the readout phase, the control unit sends control signals to the camera subsystem to read the image. The image is send to the processing unit.

[00365] In processing phase, the raw image from the camera is split into 9 different sub-images. This images have the same structure as superpixels. First sub-images consists of image values captured by the pixels in first pixel set. The second sub-image consists of image values captured by the pixels of second pixel set,...., the ninth sub-image consists of image values captured by the pixels of the ninth pixel set. All nine sub-images have the same resolution. The sub-image matrix P is an image, where every pixel have 9 values from particular sub-images (vi, ... , V9). The value of the sub-image matrix is an 9D intensity vector (the components of the vector corresponds to specific sub-image pixel value). Because pixels in a superpixel are in proximity, the individual pixels in the superpixel captures the light from approximately the same area. The centers of the superpixels are treated as centers of pixels in sub-image matrix (for purposes of geometric computations).

[00366] Each pixel in sub-image matrix P captures light from a part of the scene (either part of the object of interest, or the background of the area of interest). If the illumination module does not illuminate the area of interest, the pixel captures no light, other than any ambient light. If the illumination module illuminates the area of interest, it illuminates the area in unknown time t (it is in general a very short period of time due to rotation of the deflection mirrors). When this happens during exposure during a camera frame, t G Ti. In time period Ti, the response of the pixel sets of the image sensor are modulated by i-th codeword. Thus, the light hits the area of interest in time period Ti. In this period, only a portion of the pixel sets in the active region are "on" based on the bits of the particular codeword. The 9D intensity vector of P is (vi, v ₂ , ...., vs). Intensity values that correspond to pixel sets that are "on" during Ti are higher that a threshold (while they receive the reflected illumination from the area of interest). Intensity values that correspond to pixel sets that are "off capture no light, so the values are below the threshold. This binarization by the threshold creates an 9-bit codeword. By decoding this codeword (e.g., GrayToBinary in Fig. 4), a computer calculates the index of the time interval i (Timeslot). Thus, this algorithm decodes the time t - with the accuracy of up to T / 512 (as the center of the timeslot Ti). The codeword 000000000 is reserved for pixels that was not lit by the illumination subsystem. The 3D coordinate of (0, 0, 0) is sent to output for such pixels. (Keep in mind that, due to curtain modulation which is simultaneously occurring, the entire inactive region of the image sensor is not sensitive to light, and thus the superpixel modulation described above in this paragraph may have a practical effect only in the active region of the image sensor.)

[00367] For the time t, the processing units computes the plane equation of the vertical plane that (i) is parallel to the laser beam, (ii) intersects the laser beam, and (iii) corresponds to the particular angle a of the first deflection mirror and the particular angle β of the second deflection mirror. Using this plane equation and data gathered during calibration of the camera subsystem, a computer computes the intersection of the vertical plane and the pixel ray for the superpixel. The pixel ray is the half-line along which light from the scene travels to the center of the superpixel. The intersection is a 3D coordinate in world coordinate system, and it is saved as an output value for the particular pixel P. The algorithm is executed for all pixels of the sub-image matrix.

[00368] In output phase, the control unit sends the computed 3D coordinates out of the system.

[00369] In an illustrative embodiment of the system, the projector space

correspondence is encoded into code for pixel modulation. Thus, the 3D coordinate is computed for every pixel of sub-image matrix separately. [00370] Figure 29A is a diagram of a 3D scanning system, in an illustrative

implementation of this invention. The system includes a camera subsystem 2901 and an illumination subsystem 2903.

[00371] In some implementations of the system, the illumination subsystem 2903 comprises a laser 2921, two galvanometer-actuated optical scanners (galvos), and lens 2913. The first galvo comprises a first galvanometer actuator 2925 and a first mirror 2929. The second galvo comprises a second galvanometer actuator 2945 and a second mirror 2949. During scanning, the first galvanometer actuator 2925 controls the orientation of the first mirror 2929 and the second galvanometer actuator 2945 controls the orientation of the second mirror 2949. Together, the two gyros control the angle of deflection of a beam of laser light 2928 from the laser 2921. Lens 2913 may be used, for example, to focus a beam. However, lens 2913 is optional. In many cases, illumination subsystem 2903 does not include a lens.

[00372] In the example shown in Figure 29A, a beam of laser light 2928 is projected toward the scene 2929. The beam of laser light 2928 projects a dot of laser illumination on the scene. The scene is within both the illumination subsystem's field of view 2931 and the camera subsystem's field of view 2933. The projected light is reflected 2935 from the objects in the scene or surrounding walls toward the camera subsystem.

[00373] In the example shown in Figure 29A, the illumination subsystem projects the beam of laser light into its field of view. The projected light 2928 is reflected from the scene 2929, and passes though the camera optics 2907. The camera optics 2907 include a filter 2909 and a lens 2911. The reflected light 2935 then falls onto image sensor 2905. The laser light 2928 traverses the same path through the scene during each camera frame. As the laser light 2928 traverses this path during each camera frame, the laser light 2928 illuminates different parts of the scene at different times during each frame, such that each part of the scene is illuminated at least once during each frame.

[00374] Figure 29B is a diagram that shows four examples of light paths that may result from two rotating mirrors, in illustrative implementations of this invention. In the example shown in Figure 29B, a first mirror 2952 rotates about axis of rotation 2962 and a second mirror 2953 rotates about axis of rotation 2963. Axes 2963 and 2963 may be perpendicular to each other (e.g., horizontal and vertical). By rotating the two mirrors 2952, 2953 to different combinations of rotational positions, the two mirrors, taken together, may deflect light from laser 2951 in different directions in a 3D volume. For example, rotating the set of two mirrors 2952, 2953 may cause a light beam from laser 2951 to deflect from the mirror set in directions 2971, 2972, 2973, 2974, each at a different time.

[00375] In illustrative implementations, an illumination system (e.g., 2903 in Figure 29A and 2951, 2952, 2953 in Figure 29B) may be employed for curtain modulation. For example, the illumination system may project a beam of light in different directions in a scene at different times, such that a dot of light moves over the scene (e.g., in a zig-zag path) for curtain modulation.

[00376] In some embodiments, a mechanical deflection system is used to change the direction of the projected light. For example, in some cases, the deflection system comprises a galvanometer-actuated optical scanner, MEMS mirror or polygonal mirror. In some cases, a sensor for detecting the angle of rotation of the mirror is included in the illumination subsystem. In some cases, control signals for the reflective surfaces (e.g., two mirrors or mirrors in a DMD) specify particular positions of the reflective surfaces. Alternately, a set of mirrors (e.g., two mirrors) each oscillate, and control signals for the mirrors specify a periodic oscillation of position of the mirrors.

[00377] Figure 30 is a flowchart that shows steps in a method of 3D scanning. In the example shown in Figure 30, desired timing parameters (such as the number of frames per second) are set. (Step 3001). The timer is activated; it triggers the frame capture at the desired frame rate. (Step 3003). The system sends signals to the control unit (Step 3005). The control unit 3017 has a timer which is started at the beginning of each frame (Step 3019). According to the time provided by the control unit timer, the control unit computes the superpixel modulation for each of the pixel sets (Step 3025) and computes the position information for the illumination subsystem (Step 3021). Superpixel modulation signals are sent to the respective pixel sets (Step 3027). For curtain modulation, a computer calculates adjustments to the rolling curtain window - i.e., calculates changes in position of the active region of the image sensor. These adjustments cause the active region to move repeatedly over the image sensor, such as by making repeated vertical up movements (Step 3028). A positioning signal is sent to the illumination subsystem. For example, this positioning signal may control actuators for rotating mirrors, to control the angle at which a laser beam is deflected, and thus to control the position of a dot of laser light that moves rapidly over the scene for curtain modulation. (Step 3023). At the end of the frame (which is determined by the timer to be in sync with the desired frame rate), a processing unit receives data from the camera (Step 3007). Using the data from the camera 3009 and previously gathered calibration data 3011, a computer calculates a 3D model (Step 3013) and sends it as output (Step 3015).

[00378] Figure 31 is a flowchart that shows steps in a method of 3D scanning, in which a timeslot is calculated. In the example shown in Figure 31, the steps of the method include frame start (Step 3101), illumination (Step 3103), recording intensity 3105, reading values from camera pixels (Step 3107), comparing to thresholds, producing code, decoding (Step 3111), obtaining a timeslot and index (Step 3113), for each given distance at which dots are placed, calculating equation of a plane that best fits a set of dots photographed during various timeslots (Step 3115), computing pixel ray equation, which is the equation of a ray from the center of the camera pixel to the scene 3109, and computing an intersection of these two mathematical equations 3117.

[00379] In illustrative implementations, Figure 32 is a flowchart that shows steps in a method of calibrating the imaging system, in an illustrative implementation of this invention. The method includes: As a preliminary step, place the camera and projector units into position so as to mechanically align the camera and projection unit as precisely as possible. In some cases, also rigidly attach a temporary second camera to the imaging system. The temporary second camera may be attached adjacent to the projection unit (e.g., above or below illumination subsystem 2903 in Figure 29A), in order to provide a baseline for calibration. Then calibrate parameters of a first camera 3201. Calibrate parameters of a second camera 3203. Calibrate camera pair distances 3205. Use a galvo-actuated light source to project according to single dot parameters 3207. Calibrate position of the projection unit and cameras and calibrate parameters of the projection unit 3209. Verify parameters (e.g., by 3D scanning a flat surface) 3211. In Figure 32, the projection unit projects a moving beam of laser light onto a scene. For example, the projection unit may comprise illumination subsystem 2903 in Figure 29A or laser 2951 and mirrors 2952, 2953 in Figure 29B.

[00380] For example, in some cases, a calibration method includes the following steps. First, parameters of each single camera are determined using a calibration pattern. A precisely printed or etched calibration pattern is placed at different locations in the view of camera. The locations evenly cover the desired range and field of view of the camera. In many cases, the camera takes at least 10 pictures of the calibration pattern. Then a computer runs algorithm on this input to calculate the matrix in standard format, containing the determined parameters of a camera. If the mathematical uncertainty indicated by the algorithm is not below a desired threshold, the process is repeated.

[00381] This is repeated for all cameras, one by one.

[00382] After all cameras are calibrated, the positions of the cameras relative to each other are determined. One camera, the first one (#1), is selected as the primary reference point from which the relative position of other cameras is determined. For each pair of camera #1 - camera #x, at least 10 pictures of the calibration pattern, shot from both cameras, are taken. The pattern is placed in different location, facing the cameras, so as to evenly cover the desired range and field of view of the cameras. An algorithm takes the images as an input and computes the relative positions of the cameras. If the mathematical uncertainty indicated by the algorithm is not below a desired threshold, the process of determining the positions of the cameras is repeated with newly taken images.

[00383] After this, the position and projection parameters of the light source(s) are determined. A flat white object is placed in front of the system, so that the light from the source is reflected and the image of the projected dot can be taken by the cameras. The light source projects a number of single-dot patterns, covering its whole range. The flat white object is then placed in a different distance and the process is repeated. The flat white object is placed in at least 5 positions at different distances so as to evenly cover the desired range of the system.

[00384] Images of the patterns are provided as input for algorithm computing the relative position of the projection unit, and the parameters of the projection (mapping of numeric inputs given to projection unit to the real angle of projected line). If the

mathematical uncertainty indicated by the algorithm is not below a desired threshold, the process of determining the position of the projection unit and its parameters is repeated with newly taken images.

[00385] After this, a verification step is done. The images collected in the previous step are provided to the triangulation algorithm which computes the position of the projected dot. A plane is mathematically computed to be the best fit for the dots taken while the flat white object is at one distance. A next position of the flat white object produces another plane, etc.

[00386] In this method, if the system uses one camera, an additional second camera is temporarily, but rigidly attached to the system. The additional camera is detached after the calibration process is accomplished [00387] Here is another non-limiting example of a calibration technique, used in some implementations of this invention. In this example, the calibration parameters consist of an internal part and an external part. The internal properties of the camera include local behavior of projection, the camera matrix and the parameters of lens distortion model. The external properties include the position/rotation of the camera in respect to world coordinate system.

[00388] All parameters are calculated by an automatic or semiautomatic process. In this process, the cameras captures known calibration pattern (e.g. Figure 33) in multiple different positions/orientations (Figure 34). The automatic algorithm computes the position of the individual circles of the pattern. Each individual circle has encoded the position of its center in pattern coordinate space (in metric coordinates).

[00389] When the system knows the internal parameters of the camera and the position/orientation of the calibration pattern, the position of each individual center of circle (in world coordinate space) is projected to camera coordinate space (projected coordinate). Position of each circle is computed (image coordinate). The reprojection error is the difference between projected coordinate and the image coordinate for each circle.

[00390] At the beginning of the process, the internal parameters of the camera and the position/orientation of calibration patterns are unknown. A computer calculates the parameters using an optimization technique to minimize reprojection error for all captured images of calibration pattern (e.g. mean square error).

[00391] After the internal parameters of all cameras are calculated, the same captured patterns are used to determine the position of individual cameras with the optimization technique to minimize reprojection error.

[00392] Alternatively, other calibration techniques may be used, such as mechanical alignment of components to exact locations.

[00393] Figure 33 is a diagram that shows four examples of circles used in an optical calibration pattern. In some cases, an optical calibration pattern comprises a large number of such circles. Each circle in calibration pattern (e.g., 3301, 3303, 3305, 3307) is encoded with a unique visual code, and thus each circle in the calibration pattern is uniquely identifiable.

[00394] Figure 34 is a diagram that shows use of a calibration pattern. A calibration pattern is precisely printed or etched on a planar surface. The calibration pattern is placed at different locations 3402 (one location at a time) in the view of a camera 3401. The locations evenly cover the desired range and field of view of the camera. In some cases, the camera captures 10 or more pictures of the calibration pattern. In Figure 34, nine dots (three dots 3407, three dots 3408 and three dots 3409) indicate positions of dots of laser light that illuminated the calibration pattern, one dot at a time, when the calibration pattern was placed at different distances. (In many cases, photos are captured during calibration for more than nine dot positions).

Coordinate System

[00395] In illustrative implementations, computations for 3D scanning involve use of coordinate systems. A wide variety of coordinate systems may be used.

[00396] Here is a non-limiting example of a coordinate system that is used in some implementations of this invention, for purposes of triangulation and calibration. In illustrative implementation, the image sensor consists of plurality of pixels (e.g. thousands of pixels, or millions of pixels, or billions of pixel, or any other number). The pixels are arranged in 2D matrix, forming an 2D coordinate system (u, v), which is the image coordinate system. Every position on the image sensor is described by 2 real numbers (u, v) - 2D vector from R ² . The coordinates lie in interval (<0, U-Resolution, <0, V-Resolution>), where U-Resolution is the horizontal number of pixels in the image sensor, and V-Resolution is the vertical number of pixels in the image sensor.

[00397] In some cases, the image plane is a planar surface and the coordinate system consists of 2 orthogonal axes U and V. In another implementation of the invention, the image sensor is a curved surface.

[00398] Light reflects from the objects in the scene, passes through camera optics (including, in many case, a lens) and forms an image on the sensor surface. The image captured by the image sensor has the same coordinate system as the image sensor. The camera has a camera center and the image plane is in front of the camera center. The perpendicular distance between camera center and image plane is equal to the focal distance) of the camera. The line perpendicular to the image plane, which crosses the camera center is the principal axis. The intersection of the principal axis with the image plane is called the principal point (p).

[00399] The camera coordinate system is a local Cartesian coordinate system with 3 orthonormal axes. The origin of the coordinate system is in the camera center c. The x-axis has the same direction as the sensor's u-axis, the y-axis has the same direction as v-axis and the z-axis has the same orientation as principal axis). Each camera subsystem has its own Camera coordinate system. The coordinate system of the scene, the World Coordinate System, is a global coordinate system. The coordinate transforms between any camera coordinate system and the world coordinate system are standard coordinate conversions (represented by a coordinate conversion matrix and its inverse transform). The camera coordinate system is the same as camera subsystem coordinate system. The coordinate conversion matrix ? _t is

where, r _{i ;} - is rotation parameter, and t _t is translation parameter.

[00400] The projection from world coordinates to image coordinates is performed as follows. The world coordinate of a point in the scene is Xh ⁼ (x, y, Z, 1) ^T - in

homogeneous coordinates. World coordinates are converted to camera coordinate space by matrix multiplication, as follows:

[00401] Point x is projected to a normalized image plane (which is located at a focal distance of 1 from the camera center). The coordinates of Point x in the scene, as projected to the normalized image plane, are

x _n = 77 * x. where IT = C 0 1 0.

[00402] To convert to image coordinates xi, the xn is multiplied by a Camera Matrix A (which represents camera intrinsic parameters)

[00403] In some implementation of the system, a transformation E (mapping 2D homogeneous space to 2D homogeneous space) compensates (at least partially) for optical distortion of the lens system. The image coordinate is computed as follows.

*i = A * e[n * (R _t * Xh )] [00404] Having specific location of the pixel χ,·, there exist a bijection between image coordinates of a point x _; and a corresponding normalized coordinates X«, (coordinates on normalized image plane). Having the normalized coordinates X«, there exists an infinite number of points, each projecting to X« on the normalized image plane. All of these points lie on a half-line (ray). The equation of the half line is

X = C + (X _n - C) * d

[00405] where x is a vector of coordinates of a 3D point in space, c is a vector of a coordinates of the center of the camera (3D point), X« is a vector of normalized coordinates of the point X; and d > 0 is a real number equal to the distance (parameter). The symbol "*" is the multiplication of vector by scalar.

[00406] This relation between 2D image coordinate and the 3D half-line is called image point/coordinate re-projection.

[00407] In some alternative implementations of the invention, a more complex geometrical model is used. In this model, the projection of a pixel into space is treated as a distribution function of possible light participation on a final pixel intensity.

[00408] In some cases, a 3D scanning system employs triangulation to determine depth and employs a moving dot of laser light to illuminate the scene. In such a laser triangulation system, the calibration is sometimes performed as follows. The calibration is done in iterations. At first, the method for calibrating the camera subsystem(s) is performed as stated before. For calibrating the surface in the space, which is lit by the laser, calibration pattern comprising of multiple circles (such as 3301, 3303, 3305, 3307) is used. In the process, each of the camera subsystem captures images of the calibration pattern (the white areas comprise a diffuse surface with albedo near 1, the black areas comprise a diffuse surface with albedo near 0.5).

[00409] The calibration pattern is printed on a stable planar surface (e.g. on top of glass table). During the exposure of the camera, the mutual position of the camera

subsystem(s), the laser and the calibration pattern is fixed. At first, the camera(s) captures the image of a calibration pattern illuminated by ambient light, while the laser is off.

[00410] From this frame, for every camera, the algorithm computes the equation of plane, which corresponds (during this step of calibration) to the planar surface of the calibration pattern (This part of the process is a sub-part of the camera calibration). [00411] The position/orientation of all elements of the system stays fixed, while camera(s) take image of the pattern lit by the laser (the laser is turned on before the second exposure). The laser light projects a moving dot of laser light on the calibration pattern. An algorithm analyzes captured images to detect the trajectory of these dots (e.g., by using thresholding and by finding the center of gravity of the intensity values). The trajectory of the dots provides information regarding multiple positions of the calibration pattern. The representation of the trajectory of the dots may be a continuous chain of 2D image coordinates. The algorithm may re-project these coordinates into 3D, which gives a set of half-lines directing from camera center.

[00412] After that, the algorithm computes the 3D intersections of half-lines with the plane of the calibration patterns. These 3D coordinates are stored in memory as the chain (sample), which is a 3D representation of the projection of the laser light. The operator then changes the position of the calibration pattern (in some implementations, a robotic arm automatically changes the position of the calibration pattern), so that the new position of the calibration pattern defines a plane, that is different from all previous. During the calibration process, the operator/robot places the calibration pattern to multiple positions to ensure homogeneous coverage of measured samples in the measurement range of the system. After a sufficient number of samples is captured (at least 2),a geometrical representation of the laser illuminated 3D surface (e.g. ruled surface) is computed by minimizing the error (e.g. mean square distance) of measured sample points and the geometrical representation of the surface.

Tri angulation

[00413] In some implementations, the system computes the 3D position of the surface points in the field of view of the camera subsystem. For every pixel of the camera, the algorithm computes a pixel ray. The pixel ray is the re-projection (the half-line from the camera center). To compute the exact 3D position along the half-line, the algorithm computes d in the equation using an additional constraint. Specifically, the additional constraint is that the plane and the half-line line going from the camera center through the pixel are never parallel, i.e. there is always an intersection (as discussed below). In some implementations, the distance d is computed directly and substituted into the equation. In some

implementations, the distance d is computed from triangulation principle. The additional constraint is either a plane or second half-line where the specific point lies. [00414] In some cases, for the plane as additional constraint (the plane and the pixel's re-projection can not be parallel), the 3D point x for a specific pixel p has to satisfy two criteria:

X = C + (X _n - C) * d

where lip, is a 3D normal vector of the plane, d _p is the Euclidian distance between the plane and the origin of the coordinate system and symbol "." is the dot product of two 3D vectors. To get the exact 3D point x, the algorithm performs an algorithm to compute unknown d:

d = (n _p . c + dp) / (np . (c - x _n ))

where symbol "/" is the division of real numbers.

Because the line and the plane are not parallel, it follows that (lip . C - lip . X _n ) is nonzero, and that d exists. A computer calculates x if the d is > 0.

X = C + (Xn - C) * (lip . C + dp) / (lip . (c - Xn))

[00415] In some implementations, 3D position of a surface points in the scene is retrieved using laser light, camera and triangulation technique. For example, the system shown in Figure 28 uses laser triangulation. Laser light emits from the illumination subsystem 2801 in form of laser line(s), laser dot(s) or different light shape (e.g. Ruled surface in 3D). The mutual position and orientation of every illumination subsystem and every camera subsystem is known in any time t. In time t, the illumination subsystem emits light on the scene 2811. For every pixel of the camera, if the laser illuminates a part of the scene that is observed by the pixel, the processing unit 2805 computes the 3D position of the illuminated part of the scene. All points illuminated by the laser lies on a defined 3D surface in the scene. The pixel of the camera collects light from a ray. The 3D point lies on intersection of this ray and the surface defined by the laser. In some cases, there is more that one such intersection and the processing unit chose the final 3D position by other

regularization criteria.

[00416] In some implementations of laser triangulation, the simple equation of plane is chosen as the representation of the laser illuminated 3D surface. In this case, the algorithm for plane fitting (based on principal components) is used.

[00417] In some implementations of laser triangulation, the mutual position and orientation of the laser and the camera does not change over time. However, the position, and orientation of the laser source could change in time in a well controlled manner (e.g.

mechanical change in position, rotation, deflection by mirror). In this case, the calibration is performed for fixed number of different laser source positions/orientations. The equation of the laser illuminated surface for all other positions/orientations are computed using sufficient mathematical model and interpolation.

[00418] In many implementations of tri angulation, profile inspection is performed. In that case, the illumination subsystem comprises the laser line source. During the

measurement, the laser light illuminate single profile of the scene, the camera subsystem captures the projection of the laser light and the algorithm analyze the image with the approach used in part of calibration. The 3D points represents a profile of the measurement.

Multiple Camera Subsystems

[00419] In some implementations, the 3D scanning system includes two camera subsystems. For each camera and each pixel, the algorithm computes time t of the light peak. The values are stored in time images (the pixel of the images are the values of t).

[00420] The first camera subsystem is primary. For every pixel of the primary camera, the algorithm computes the pixel re-projection. The corresponding 3D points lies on epipolar lines in the image of the second camera. On the time image of the second camera, the epipolar line cuts the image, which creates a ID function of time values. The algorithm computes any value on the epipolar line using bilinear interpolation from adjacent pixels. The corresponding projection of the surface point lies on the epipolar line and have the same time t, as the pixel form the primary camera. The algorithm computes the position of the value t in the ID function (with subpixel accuracy using bilinear interpolation). This position corresponds to pixel position on the second camera. The algorithm re-projects the pixel position as a half-line in 3D. The algorithm computes the 3D point as an intersection of two half-lines. An advantage of this approach is higher robustness and accuracy, while the geometry calculations are not dependent on calibration of Illumination subsystem

Alternative Implementations

[00421] In some implementations of the system, the system comprises plurality of illumination subsystems. In some implementations of this invention, the features in the scene are formed artificially by an illumination subsystem. There are different coding strategies that create specific coding for different points in the scene. This allows for more robust and accurate feature detection and position estimation.

[00422] In some implementations, this invention operates in multiple modes. One such mode is a high spatial resolution mode, which uses the same modulation signal for all pixels. The pixels from different sets are combined to produce one high spatial resolution image. In this mode, all pixels in the image sensor act like one pixel set. The ability to switch to this mode allows the scanner to select between real-time (lower resolution) mode and non-real-time (multiple sequential camera frames) high resolution mode for a more precise reconstruction.

[00423] In some implementations of the system, the system operates either in realtime mode (with different pixel modulation signals), or in high-res mode. In high-res mode, the scanning process may comprise multiple exposures and laser sweeps in succession. In every exposure during high-res mode, the control units apply the superpixel modulation signals to whole image sensor. The multiple pixel values consist of values captured in succession. This allows to compute the time t of the light peak and appropriate 3D point for every pixel in the image sensor.

[00424] In some implementations of the invention, the system uses different coding strategies to encode the time of pixel illumination alongside Gray codes. In some

implementations, the control units creates a set of periodic sinusoidal functions with equal period and different phase shift (e.g. 4 sinusoidal functions shifted by π/2 rad each after another). The set of modulation function (phase shifting modulation functions) is applied to appropriate number of image sets (in some cases, multiple image sets are modulated by the same function to collect more samples). The corresponding pixel sets (and the corresponding pixels in every superpixels) are phase shifting pixels (for the simplicity of description). The number of periods of the modulation functions increase the possible accuracy of the coding. In many cases, the period of the function is at least 2 times longer that the time period T _s the laser line needs to swipe through the area of projection of individual pixels of the sensor.

[00425] In some implementations, the distance to measured surface does not affect time Ts. While the time period T _s is short (due to given constraints), the received illumination (the collected pixel intensity), the values of phase shifting pixels in superpixels, encodes the phase (the position inside the period). While the coding uses periodic function, the algorithm computes the index of period from the Gray coded pixels (the index time period Ti is shorter than the period of the used sinusoidal functions). In some implementations, the system uses 4 sinusoidal functions shifted by nil rad each after another. The values of corresponding phase shifting pixels are AO, Al, A2, A3. The algorithm computes the phase value φ using the following equation:

φ = atan2(A0 - A2, Al - A3) arciaa ~f~ π v > G.a? < 0

arc tao V < I), J < !

+|

"5? ^' iEcte toac! y = ti> %— I

[00426] The time t of the light peak may be calculated as:

t = Periodlndex * T _pe riod + [(φ + π) I (2 * π)] * T _pe nod where Tperiod is the length of the period of the sinusoidal function, and the Periodlndex is the index of the period derived from time period Ti.

[00427] In some implementations, the system uses phase shifting modulation functions with different period. For example, first set of phase shifting modulation signal have period of the same length as the exposure T, the second set has period of T/256. The algorithm decodes the phase value of the light peak (the time, when the laser dot moves across the projection of the individual pixel of the sensor) for both sets. The second phase value represents accurate measurement of the time of the occurrence of the light peak (up to the period index), the first phase value represents unambiguous measurement (Because of the physical limitations of the sensor, such measurement is in the most cases less accurate, than the one from multiple periods phase shifting modulation functions), which the algorithm uses to set the right period index.

[00428] In some implementations, the system uses different modulation functions (e.g. square waves).

[00429] In some implementations of the system, each pixel stores charge in multiple storage devices. The modulation signals controls the allocation of the accumulated photo- electrons to different storages. This allows for capturing multiple phase shifting values in single pixel. Computers

[00430] In exemplary implementations of this invention, one or more electronic computers (e.g. 2803, 2805) are programmed and specially adapted: (1) to control the operation of, or interface with, hardware components of an imaging system, including by modulating the transfer of charge in a pixel photodiode to one or more storage devices, and for controlling a reflective or transmissive SLM to control intensity of light incident on image sensor pixels; (2) to control the operation of, or interface with, hardware components of a 3D scanning system, including an illumination subsystem and a camera subsystem; (3) to process data captured by an image sensor; (4) to perform triangulation to reconstruct 3D surface geometry; (5) to perform calibration, including calibration of any camera or light source; (6) to output modulation signals for modifying the response of pixels to incident light; (7) to output control signals to synchronize illumination of a scene with modulation of pixels; (8) to perform any other calculation, computation, program, algorithm, computer function or computer task described or implied above, including any optimization algorithm; (9) to receive signals indicative of human input; (10) to output signals for controlling transducers for outputting information in human perceivable format; and (11) to process data, to perform computations, to execute any algorithm or software, and to control the read or write of data to and from memory devices. The one or more computers may be in any position or positions within or outside of a 3D scanning system or imaging system. For example, in some cases

(a) at least one computer is housed in or together with other components of the system, and

(b) at least one computer is remote from other components of the system. The one or more computers are connected to each other or to other components in the system either: (a) wirelessly, (b) by wired connection, or (c) by a combination of wired and wireless connections.

[00431] In exemplary implementations, one or more computers are programmed to perform any and all calculations, computations, programs, algorithms, computer functions and computer tasks described or implied above. For example, in some cases: (a) a machine- accessible medium has instructions encoded thereon that specify steps in a software program; and (b) the computer accesses the instructions encoded on the machine-accessible medium, in order to determine steps to execute in the program. In exemplary implementations, the machine-accessible medium comprises a tangible non-transitory medium. In some cases, the machine-accessible medium comprises (a) a memory unit or (b) an auxiliary memory storage device. For example, in some cases, a control unit in a computer fetches the instructions from memory.

[00432] In illustrative implementations, one or more computers execute programs according to instructions encoded in one or more a tangible, non-transitory, computer- readable media. For example, in some cases, these instructions comprise instructions for a computer to perform any calculation, computation, program, algorithm, computer function or computer task described or implied above. For example, in some cases, instructions encoded in a tangible, non-transitory, computer-accessible medium comprise instructions for a computer to: (1) to control the operation of, or interface with, hardware components of an imaging system, including by modulating the transfer of charge in a pixel photodiode to one or more storage devices, for controlling a reflective or transmissive SLM to control intensity of light incident on image sensor pixels; (2) to control the operation of, or interface with, hardware components of a 3D scanning system, including an illumination subsystem and a camera subsystem; (3) to process data captured by an image sensor; (4) to perform

triangulation to reconstruct 3D surface geometry; (5) to perform calibration, including calibration of any camera or light source; (6) to output modulation signals for modifying the response of pixels to incident light; (7) to output control signals to synchronize illumination of a scene with modulation of pixels; (8) to control and synchronize movement of a dot of light (which moves across a scene) and movement of an active region of an image sensor; (9) to perform any other calculation, computation, program, algorithm, computer function or computer task described or implied above, including any optimization algorithm; (10) to receive signals indicative of human input; (11) to output signals for controlling transducers for outputting information in human perceivable format; and (12) to process data, to perform computations, to execute any algorithm or software, and to control the read or write of data to and from memory devices.

Definitions

[00433] The terms "a" and "an", when modifying a noun, do not imply that only one of the noun exists.

[00434] To compute "based on" specified data means to perform a computation that takes the specified data as an input.

[00435] "Boolean logic" means Boolean algebra. [00436] Here are some non-limiting examples of a "camera" : (a) an optical instrument that records images; (b) a digital camera; (c) a video camera; (d) a camera that uses photographic film or a photographic plate; (e) a light field camera; (f) an imaging system, (g) an imaging sensor; (h) a time-of-flight camera; (h) apparatus that includes an imaging sensor or an array of imaging sensors; and (i) apparatus for gathering data about light incident on the apparatus. The term "camera" includes any computers that process data captured by the camera.

[00437] The term "comprise" (and grammatical variations thereof) shall be construed as if followed by "without limitation". If A comprises B, then A includes B and may include other things.

[00438] The term "computer" includes any computational device that performs logical and arithmetic operations. For example, in some cases, a "computer" comprises an electronic computational device, such as an integrated circuit, a microprocessor, a mobile computing device, a laptop computer, a tablet computer, a personal computer, or a mainframe computer. For example, in some cases, a "computer" comprises: (a) a central processing unit, (b) an ALU (arithmetic/logic unit), (c) a memory unit, and (d) a control unit that controls actions of other components of the computer so that encoded steps of a program are executed in a sequence. For example, in some cases, the term "computer" also includes peripheral units, including an auxiliary memory storage device (e.g., a disk drive or flash memory). However, a human is not a "computer", as that term is used herein.

[00439] "Defined Term" means a term or phrase that is set forth in quotation marks in this Definitions section or in the claims.

[00440] A "dot of light" that is projected on a scene means collimated light that is projected on a scene. The term "dot of light" does not imply any particular shape or size.

[00441] The term "e.g." means for example.

[00442] The fact that an "example" or multiple examples of something are given does not imply that they are the only instances of that thing. An example (or a group of examples) is merely a non-exhaustive and non-limiting illustration.

[00443] Unless the context clearly indicates otherwise: (1) a phrase that includes "a first" thing and "a second" thing does not imply an order of the two things (or that there are only two of the things); and (2) such a phrase is simply a way of identifying the two things, respectively, so that they each can be referred to later with specificity (e.g., by referring to "the first" thing and "the second" thing later). For example, unless the context clearly indicates otherwise, if an equation has a first term and a second term, then the equation may (or may not) have more than two terms, and the first term may occur before or after the second term in the equation. A phrase that includes a "third" thing, a "fourth" thing and so on shall be construed in like manner.

[00444] The term "for instance" means for example.

[00445] In the context of a camera (or components of the camera), "front" is optically closer to the scene being imaged, and "rear" is optically farther from the scene, during normal operation of the camera.

[00446] "Herein" means in this document, including text, specification, claims, abstract, and drawings.

[00447] Unless the context clearly indicates otherwise, "high" and "low" are Boolean logic values. For example, "high" and "low" Boolean logic values may be physically represented by particular voltage ranges. Which voltage range is assigned to which Boolean logic value is arbitrary. Thus, for example: (a) the Boolean logic value "high" may be represented by a first voltage range; (b) the Boolean logic value "low" may be represented by a second voltage range; and (c) voltages in the first voltage range may be greater than voltages in the second voltage range, or vice versa

[00448] The term "image sensor" is another way of saying "imaging sensor".

[00449] "Imaging pixel" means a component of an image sensor, which component takes measurements of intensity of light incident on a region of the image sensor, such that the intensity is measured for the entire region and is not separately measured for any subregion of the region.

[00450] "Imaging superpixel" means a set of imaging pixels, such that each imaging pixel in the set is a horizontal, vertical or diagonal neighbor of another imaging pixel in the set.

[00451] As used herein: (1) "implementation" means an implementation of the present invention; (2) "embodiment" means an embodiment of the present invention; (3) "case" means an implementation of the present invention; and (4) "use scenario" means a use scenario of the present invention.

[00452] The term "include" (and grammatical variations thereof) shall be construed as if followed by "without limitation".

[00453] "Intensity" means any measure of or related to intensity, energy or power. For example, the "intensity" of light includes any of the following measures: irradiance, spectral irradiance, radiant energy, radiant flux, spectral power, radiant intensity, spectral intensity, radiance, spectral radiance, radiant exitance, radiant emittance, spectral radiant exitance, spectral radiant emittance, radiosity, radiant exposure or radiant energy density. In the case of a ratio of two intensities, both of the intensities in the ratio have the same SI units.

[00454] "Light" means electromagnetic radiation of any frequency. For example, "light" includes, among other things, visible light and infrared light. Likewise, any term that directly or indirectly relates to light (e.g., "imaging ") shall be construed broadly as applying to electromagnetic radiation of any frequency.

[00455] The phrase "logical AND" of X and Y means a Boolean value that is equal to X AND Y, where "AND" in capital letters means the Boolean algebra operator AND.

Y

[00456] To say that X is "much less" than Y means that - > X.

[00457] The term "n-ary" is defined elsewhere in this document.

[00458] Two pixels are "neighbors" if the two pixels are positioned such that (i) the two pixels are adjacent to each other and (ii) no pixel is between the two pixels.

[00459] The term "or" is inclusive, not exclusive. For example A or B is true if A is true, or B is true, or both A or B are true. Also, for example, a calculation of A or B means a calculation of A, or a calculation of B, or a calculation of A and B. Nothwithstanding the foregoing, the word "or" in the phrase "high or low" is an exclusive or.

[00460] A "read-out frame" means a period of time that begins when an exposure of an imaging pixel to light starts and that ends when digital data is created that is indicative of charge accumulated during the exposure.

[00461] To say that a pixel is "receiving" light at a given time means that the light (i) is incident on the pixel at the given time or (ii) would be incident on the pixel at the given time, but for attenuation by an SLM.

[00462] "Reflective SLM" means a device that (i) reflects light from the device, and (ii) attenuates the light, such that the amount of attenuation of a light ray incident at a point on a surface of the device depends on at least the 2D spatial position of the point on the surface.

[00463] As used herein, the term "same path" is construed such that, whether light that is projected by an illumination source onto a scene travels in the "same path" through the scene in each period: (a) depends solely on the time-varying angular orientation of the light relative to the illumination source during each period; and (b) does not depend on the position of the light relative to a feature in the scene, including relative to a moving feature in the scene.

[00464] As used herein, the term "set" does not include a so-called empty set (i.e., a set with no elements) and does not include a set with only one element. Mentioning a first set and a second set does not, in and of itself, create any implication regarding whether or not the first and second sets overlap (that is, intersect).

[00465] The term "signal generator" includes any signal generator, function generator, digital pattern generator, frequency generator and any other electronic device that generates repeating or non-repeating electrical signals, including either digital or analog signals.

[00466] To say that two things are "simultaneous" means that they overlap in time, at least partially.

[00467] "Some" means one or more.

[00468] A "spatial light modulator", also called an "SLM", means an electronically controlled device that (i) transmits light through the device or reflects light from the device, and (ii) attenuates the light, such that the amount of attenuation of a light ray incident at a point on a surface of the device depends on at least the 2D spatial position of the point on the surface.

[00469] A non-limiting example of changing the "sensitivity" of an imaging pixel is to change the state of a circuit so as to control which storage device, out of a set of storage devices, receives a particular electric charge, which electric charge (i) accumulated in a photodiode of the imaging pixel and (ii) will not be and has not been transferred to or from any other storage device in the set.

[00470] To say that a set of signals is "separate" means that no signal in the set is a function of any other signal in the set.

[00471] A "storage device" means any device for storing electric charge, including a capacitor, floating diffusion or capacitive bin.

[00472] "Subregion" of a first region means a part, but not all, of the first region.

[00473] As used herein, a "subset" of a set consists of less than all of the elements of the set.

[00474] "Substantially" means at least ten percent. For example: (a) 112 is

substantially larger than 100; and (b) 108 is not substantially larger than 100.

[00475] "Substantially all" means between 98% and 100% of.

[00476] The term "such as" means for example. [00477] To say that a region of an imaging sensor "sweeps over" the sensor means that the region undergoes movement relative to the sensor, which movement (a) is in a constant direction relative to the sensor, and (b) occurs by changing which pixels are members of the region.

[00478] Spatially relative terms such as "under", "below", "above", "over", "upper", "lower", and the like, are used for ease of description to explain the positioning of one element relative to another. The terms are intended to encompass different orientations of an object in addition to different orientations than those depicted in the figures.

[00479] "Transmissive SLM" means a device that (i) transmits light through the device, and (ii) attenuates the light, such that the amount of attenuation of a light ray incident at a point on a surface of the device depends on at least the 2D spatial position of the point on the surface.

[00480] Except to the extent that the context clearly requires otherwise, if steps in a method are described herein, then the method includes variations in which: (1) steps in the method occur in any order or sequence, including any order or sequence different than that described; (2) any step or steps in the method occurs more than once; (3) different steps, out of the steps in the method, occur a different number of times during the method, (4) any combination of steps in the method is done in parallel or serially; (5) any step or steps in the method is performed iteratively; (6) a given step in the method is applied to the same thing each time that the given step occurs or is applied to different things each time that the given step occurs; or (7) the method includes other steps, in addition to the steps described.

[00481] This Definitions section shall, in all cases, control over and override any other definition of the Defined Terms. For example, the definitions of Defined Terms set forth in this Definitions section override common usage or any external dictionary. If a given term is explicitly or implicitly defined in this document, then that definition shall be controlling, and shall override any definition of the given term arising from any source (e.g., a dictionary or common usage) that is external to this document. If this document provides clarification regarding the meaning of a particular term, then that clarification shall, to the extent applicable, override any definition of the given term arising from any source (e.g., a dictionary or common usage) that is external to this document. To the extent that any term or phrase is defined or clarified herein, such definition or clarification applies to any

grammatical variation of such term or phrase, taking into account the difference in grammatical form. For example, the grammatical variations include noun, verb, participle, adjective, or possessive forms, or different declensions, or different tenses. In each case described in this paragraph, Applicant is acting as Applicant's own lexicographer.

[00482] A defined term is merely a symbol that signifies the item that is defined. Any words that appear inside the defined term itself have no effect on the definition. For example, if "red elephant" were a defined term that was defined to mean a laser, then: (a) "red" would have no effect on the meaning of the defined term (e.g. it would not imply that the laser is red or emits red light), and (b) "elephant" would have no effect on the meaning of the defined term (e.g., it would not imply that the laser is wrinkly or shoots water out of a trunk).

Likewise, in the defined terms "window signal" and "window region", the word "window" does not have any effect on the meaning of the items being defined. Likewise in the defined terms "max region" and "superpixel signal", the terms "max" and "superpixel" do not have any effect on the meaning of the defined terms.

Variations:

[00483] In many implementations of this invention, both curtain modulation and superpixel modulation are applied simultaneously. Alternatively, in some cases, curtain modulation (for ambient light suppression) is not performed concurrently with superpixel modulation. In those cases, superpixels do not undergo superpixel modulation while in the active region of the image sensor. Ambient light suppression without superpixel modulation is useful in many scenarios, including some scenarios in which the scene is illumined with light of desired color, intensity, direction, polarization or combinations of these parameters and it is desirable to record reflected light from the illumination, rather than ambient light.

[00484] This invention may be implemented in many different ways. For example:

[00485] In some implementations, this invention is a method comprising, during a period of time: (a) modulating the sensitivity of imaging pixels in superpixels of an imaging sensor by a first signal ("window signal") and simultaneously by a set of other signals ("superpixel signals"), such that (i) the sensitivity is sensitivity to light, (ii) each respective pixel, in the imaging pixels, is modulated by the window signal and simultaneously by a given superpixel signal that is one of the modulation signals, which window signal and given superpixel signal, taken together, cause the sensitivity of the respective pixel to light to vary over time; and (b) an illumination system projecting a dot of light on a scene, which dot of light moves in a path through the scene; wherein (1) during the period (i) the pixels of the imaging sensor comprise pixel sets, (ii) the superpixel signal for each pixel set is separate from the superpixel signal for each other pixel set, and (iii) each of the superpixels includes at least one pixel from each of the pixel sets, and thus there are separate superpixel signals for different pixels in each superpixel, (2) at any given time that the dot of light is projected on the scene during the period, each pixel in a first region ("max region") of the imaging sensor is receiving light that is reflecting directly from the scene, (3) during the period (A) the max region repeatedly changes position relative to the imaging sensor, due to the dot of light changing position in the scene, (B) a second region ("window region") of the imaging sensor also repeatedly changes position relative to the imaging sensor, and (C) the changes in position of the max region and window region relative to the imaging sensor occur by changing which pixels are members of the max region and window region, respectively, and (4) at any time that the dot of light is projected on the scene during the period (A) the window region includes substantially all of the pixels in the max region and also includes other pixels, (B) the number of pixels in the window region is much less than the total number of pixels in the imaging sensor, and (C) the window signal is high for all pixels in the window region and low for all other pixels in the imaging sensor. In some cases, the window region comprises: (a) a set of rows of pixels of the image sensor, such that each row in the set of rows is adjacent to another row in the set of rows; or (b) a set of columns of the image sensor, such that each column in the set of columns is adjacent to another column in the set of columns. In some cases, during the period, the window region sweeps repeatedly over the imaging sensor, such that (i) at least a portion of each row of pixels of the imaging sensor is included in the window region during a portion of each sweep, or (ii) at least a portion of each column of pixels of the imaging sensor is included in the window region during a portion of each sweep. In some cases, during each readout frame of the image sensor, the window region sweeps repeatedly over the imaging sensor, such that (i) at least a portion of each row of pixels of the imaging sensor is included in the window region during a portion of each sweep, or (ii) at least a portion of each column of pixels of the imaging sensor is included in the window region during a portion of each sweep. In some cases, the frequency of the window signal is greater than the frequency of each of the superpixel signals. In some cases, during the period, the path of the dot of light includes a series of repeated movements along trajectories that are parallel to each other. In some cases, during each readout frame of the imaging sensor, the path of the dot of light includes a series of repeated movements along trajectories that are parallel to each other. In some cases, at any time that the dot of light is undergoing movement during the period, the longest dimension of the window region is perpendicular to the direction of the movement. In some cases, a window signal that is applied to a specific pixel of the imaging sensor controls storage of electric charge that accumulates in the specific pixel. In some cases, a superpixel signal that is applied to a specific pixel of the imaging sensor controls storage of electric charge that accumulates in the specific pixel. In some cases, the window signal that is applied to a specific pixel of the imaging sensor controls attenuation of light that reaches, or would in the absence of the attenuation reach, the specific pixel. In some cases, a superpixel signal that is applied to a specific pixel of the imaging sensor controls attenuation of light that reaches, or would in the absence of the attenuation reach, the specific pixel. In some cases: (a) at any time during the period, the superpixel signal for a specific pixel is high or low and the window signal for the specific pixel is high or low, and (b) the specific pixel is more sensitive to light when both the window signal and the modulation signal are high than when at least one of the window signal and the modulation signal are low. In some cases: (a) the sensitivity to light of a specific pixel of the imaging sensor is controlled by another signal ("sensitivity signal") that is applied to the specific pixel; and (b) during the period the sensitivity signal for the specific pixel (i) is high when the window signal for the specific pixel and the superpixel signal for the specific pixel are simultaneously high, and (ii) is otherwise low. In some cases, the window signal for the specific pixel and the superpixel signal for the specific pixel control the sensitivity of the specific pixel to light only indirectly through the sensitivity signal. In some cases: (a) the sensitivity signal for a specific pixel is generated by a computer; and (b) the window signal for the specific pixel and the superpixel signal for the specific pixel are calculated by, and encoded by voltage or electric charge levels in, the computer. In some cases, the sensitivity signal for a specific pixel of the imaging sensor controls storage of electric charge that accumulates in the specific pixel. In some cases, the sensitivity signal for a specific pixel of the imaging sensor controls attenuation of light that reaches, or would in the absence of the attenuation reach, the specific pixel. In some cases: (a) the superpixel signals and the window signal (together, "modulation signals") each consist of a temporal sequence of states; and (b) the number of times that each specific state occurs, and the order in which the states occur, varies from modulation signal to modulation signal. In some cases: (a) the modulating occurs during a readout-frame, which readout-frame includes an exposure period; and (b) each imaging pixel in the imaging sensor includes a photodiode that, during the exposure period, transfers electric charge to only one storage device, such that at multiple times during the exposure period, electric charge is transferred from the photodiode to the storage device. In some cases: (a) the modulating occurs during a readout-frame, which readout-frame includes an exposure period; and (b) each imaging pixel in the imaging sensor includes a photodiode that, at multiple times during the exposure period, transfers electric charge to a set of storage devices, such that (i) at any time at which a transfer of electric charge from the photodiode to a storage device occurs, electric charge is being transferred to only one of the storage devices, and electric charge is not being transferred to any other storage device, and (ii) electric charge is transferred from the photodiode to each of the storage devices during the exposure period. In some cases, the set of superpixel signals, taken together, comprise a code, each of the superpixel signals representing a digit of the code. In some cases, a computer calculates 3D spatial coordinates of points in the scene, based on which specific permutation of the code occurs when light reflects directly from the point in the scene to the imaging sensor. Each of the cases described above in this paragraph is an example of the method described in the first sentence of this paragraph, and is also an example of an embodiment of this invention that may be combined with other embodiments of this invention.

[00486] In some implementations, this invention is a system comprising: (a) a camera that includes an imaging sensor, which imaging sensor includes (i) pixel sets, each pixel set comprising multiple imaging pixels, and (ii) imaging superpixels; (b) an illumination system that is programmed to project a dot of light on a scene, such that the dot of light moves in a path through the scene; and (c) one or more signal generators that are programmed to generate a first signal ("window signal") and a set of other signals ("superpixel signals"), such that (i) during a period of time, each respective pixel, in the imaging pixels, is modulated by the window signal and simultaneously by a given superpixel signal that is one of the modulation signals, which window signal and given superpixel signal, taken together, cause the sensitivity of the respective pixel to light to vary over time, (ii) during the period (A) the superpixel signal for each pixel set is separate from the superpixel signal for each other pixel set, and (B) each of the superpixels includes at least one pixel from each of the pixel sets, and thus there are separate superpixel signals for different pixels in each superpixel, (iii) at any given time that the dot of light is projected on the scene during the period (A) the window signal is high for all pixels in a first region ("window region") of the imaging sensor and is low for all other pixels in the imaging sensor, (B) the number of pixels in the window region is much less than the total number of pixels in the imaging sensor, and (C) the window region includes a second region ("max region") of the imaging sensor and also includes other pixels of the imaging sensor, the max region being a region that includes substantially all pixels of the imaging sensor that are receiving light that is reflecting directly from the scene at the given time, and (iv) the window region repeatedly changes position relative to the imaging sensor, as the dot of light moves through the scene and thus the max region changes position relative to the imaging sensor, which changes in position of the window region and of the max region occur by changing which pixels are members of the window region and of the max region, respectively. In some cases, the one or more signal generators are programmed to generate the window signal such that the window region comprises: (a) a set of rows of pixels of the image sensor, such that each row in the set of rows is adjacent to another row in the set of rows; or (b) a set of columns of the image sensor, such that each column in the set of columns is adjacent to another column in the set of columns. In some cases, the one or more signal generators are programmed to generate the window signal such that, during the period, the window region sweeps repeatedly over the imaging sensor, such that (i) at least a portion of each row of pixels of the imaging sensor is included in the window region during a portion of each sweep, or (ii) at least a portion of each column of pixels of the imaging sensor is included in the window region during a portion of each sweep. In some cases, the one or more signal generators are programmed to generate the window signal such that, during each readout frame of the image sensor, the window region sweeps repeatedly over the imaging sensor, such that (i) at least a portion of each row of pixels of the imaging sensor is included in the window region during a portion of each sweep, or (ii) at least a portion of each column of pixels of the imaging sensor is included in the window region during a portion of each sweep. In some cases, the one or more signal generators are programmed to generate the window signal and the superpixel signals such that frequency of the window signal is greater than the frequency of each of the superpixel signals. In some cases, the illumination system is programmed to project the dot of light on the scene such that, during the period, the path of the dot of light includes a series of repeated movements along trajectories that are parallel to each other. In some cases, the illumination system is programmed to project the dot of light on the scene such that, during each readout frame of the imaging sensor, the path of the dot of light includes a series of repeated movements along trajectories that are parallel to each other. In some cases, the illumination system and one or more signal generators are programmed such that, at any time that the dot of light is undergoing movement during the period, the longest dimension of the window region is perpendicular to the direction of the movement. In some cases, the imaging sensor is configured such that a window signal that is applied to a specific pixel of the imaging sensor controls storage of electric charge that accumulates in the specific pixel. In some cases, the imaging sensor is configured such that a superpixel signal that is applied to a specific pixel of the imaging sensor controls storage of electric charge that accumulates in the specific pixel. In some cases: (a) the camera includes one or more spatial light modulators; and (b) the one or more spatial light modulators are programmed such that the window signal that is applied to a specific pixel of the imaging sensor controls attenuation of light that reaches, or would in the absence of the attenuation reach, the specific pixel. In some cases: (a) the camera includes one or more spatial light modulators; and (b) the one or more spatial light modulators are programmed such that a superpixel signal that is applied to a specific pixel of the imaging sensor controls attenuation of light that reaches, or would in the absence of the attenuation reach, the specific pixel. In some cases: (a) the one or more signal generators are programmed such that at any time during the period, the superpixel signal for a specific pixel is high or low and the window signal for the specific pixel is high or low, and (b) the camera is configured such that the specific pixel is more sensitive to light when both the window signal and the modulation signal are high than when at least one of the window signal and the modulation signal are low. In some cases: (a) the camera is configured such that the sensitivity to light of a specific pixel of the imaging sensor is controlled by another signal ("sensitivity signal") that is applied to the specific pixel; and (b) the one or more signal generators are programmed such that, during the period, the sensitivity signal for the specific pixel (i) is high when the window signal for the specific pixel and the superpixel signal for the specific pixel are simultaneously high, and (ii) is otherwise low. In some cases, the camera is configured such that the window signal for the specific pixel and the superpixel signal for the specific pixel control the sensitivity of the specific pixel to light only indirectly through the sensitivity signal. In some cases, a computer in the one or more signal generators is programmed: (a) to calculate, and to encode by voltage or electric charge levels in the computer, the window signal for a specific pixel and the superpixel signal for the specific pixel, and (b) to calculate, based on the window signal for the specific pixel and the superpixel signal for the specific pixel, the sensitivity signal for the specific pixel. In some cases, the imaging sensor is configured such that the sensitivity signal for a specific pixel of the imaging sensor controls storage of electric charge that accumulates in the specific pixel. In some cases, the camera is configured such that the sensitivity signal for a specific pixel of the imaging sensor controls attenuation of light that reaches, or would in the absence of the attenuation reach, the specific pixel. In some cases, the one or more signal generators are programmed such that: (a) the superpixel signals and the window signal (together,

"modulation signals") each consist of a temporal sequence of states; and (b) the number of times that each specific state occurs, and the order in which the states occur, varies from modulation signal to modulation signal. In some cases, the camera is configured such that: (a) the modulating occurs during a readout-frame, which readout-frame includes an exposure period; and (b) each imaging pixel in the imaging sensor includes a photodiode that, during the exposure period, transfers electric charge to only one storage device, such that at multiple times during the exposure period, electric charge is transferred from the photodiode to the storage device. In some cases, the camera is configured such that: (a) the modulating occurs during a readout-frame, which readout-frame includes an exposure period; and (b) each imaging pixel in the imaging sensor includes a photodiode that, at multiple times during the exposure period, transfers electric charge to a set of storage devices, such that (i) at any given time at which a transfer of electric charge from the photodiode to a storage device occurs, electric charge is being transferred to only one of the storage devices, and electric charge is not being transferred to any other storage device, and (ii) electric charge is transferred from the photodiode to each of the storage devices during the exposure period. In some cases, the set of superpixel signals, taken together, comprise a code, each of the superpixel signals representing a digit of the code. In some cases, the system further comprises a computer that is programmed to calculate 3D spatial coordinates of a point in the scene, based on which specific permutation of the code occurs when light reflects directly from the point in the scene to the imaging sensor. Each of the cases described above in this paragraph is an example of the system described in the first sentence of this paragraph, and is also an example of an embodiment of this invention that may be combined with other embodiments of this invention.

[00487] The above description (including without limitation any attached drawings and figures) describes illustrative implementations of the invention. However, the invention may be implemented in other ways. The methods and apparatus which are described above are merely illustrative applications of the principles of the invention. Other arrangements, methods, modifications, and substitutions by one of ordinary skill in the art are therefore also within the scope of the present invention. Numerous modifications may be made by those skilled in the art without departing from the scope of the invention. Also, this invention includes without limitation each combination and permutation of one or more of the abovementioned implementations, embodiments and features.