Field of the Invention

This invention relates to a time of flight camera and methods of operating this time of flight camera. In preferred embodiments the invention may be used to provide and operate a stepped frequency continuous wave time of flight camera with reduced susceptibility to signal noise and errors.

Background of the Invention

Time of flight camera systems are able to resolve distance or depth information from light which has been modulated and reflected from an object in a scene. These camera systems calculate a distance measurement for objects in a scene based on information extracted from received reflected light.

One form of time of flight camera implementation employs amplitude modulated continuous wave light transmissions - AMCW. With these systems data for a single image is captured by taking measurements of received reflected light which has been modulated with a number of different phase offsets. These different phase offset values provide data which can be processed to resolve the distance between a particular target object and a receiving camera. These systems are relatively easy to implement with the signals used being computationally straightforward to process. An example of this type of AMCW time of flight range imaging technology is disclosed in the patent specification published as PCT Publication No. W02004/090568.

An alternative form of time of flight camera employs stepped frequency continuous wave light transmissions - SFCW. With this implementation data for a single image is captured by taking measurements of received reflected light which has been modulated with a number of different frequencies.

Again the use of a periodic modulation signal which changes in frequency by a regular amount provides data which can be processed to resolve the distance between a particular target object to a receiving camera sensor. Spectral analysis of this sensor data provides frequency information which indicates the range from the sensor to reflecting objects in the scene under investigation.

SFCW techniques can be utilised as an alternative to AMCW systems, and in particular applications may mitigate problems experienced in AMCW systems by phase wrapping at or past an 'ambiguity distance'. This problem is caused by AMCW techniques using phase information to determine range information, and the inability to distinguish the range of objects separated by a multiple of the wavelength of the modulation frequency used.

Conversely the resolvable range of SFCW systems is dictated by size of the frequency steps applied to the modulation signal used. The number of frequency steps is ultimately determined by the bandwidth of the sensor used in the camera. These camera systems therefore do not confuse the range of objects in the field of view of the camera and can provide accurate range information over a specific distance.

It would be of advantage to have improvements in the field of stepped frequency continuous wave time of flight camera systems which improved on the prior art or provided an alternative choice to the prior art. In particular it would be of advantage to have reduced or mitigated signal noise and errors in the measurements recorded with stepped frequency continuous wave time of flight camera systems.

Disclosure of the Invention

According to one aspect of the present invention there is provided a method of operating a time of flight camera characterised by the steps of: a) generating a source modulation signal, b) transmitting light modulated by the source modulation signal from a camera light source and capturing a time of flight camera data frame from a camera sensor illuminated with reflected light modulated by the source modulation signal, c) modifying the frequency of the source modulation signal by adding or subtracting a multiple of an offset frequency value and modifying the phase of the source modulation signal by adding or subtracting a multiple of an offset phase value to generate a stepped modulation signal, d) transmitting light modulated by the stepped modulation signal from the camera light source and capturing a time of flight camera data frame from the camera sensor illuminated with reflected light modulated by the stepped modulation signal, e) modifying the frequency of the stepped modulation signal by adding or subtracting a multiple of the offset frequency value and modifying the phase of the stepped modulation signal by adding or subtracting a multiple of the offset phase value to generate an updated stepped modulation signal, f) transmitting light modulated by the updated stepped modulation signal from the camera light source and capturing a time of flight camera data frame from the camera sensor illuminated with reflected light modulated by the updated stepped modulation signal, g) repeating steps e and f to generate further updated stepped modulation signals and capturing time of flight camera data frames using said updated modulation signals to provide a time of flight camera data set, and h) Processing the data set to determine range information for objects reflecting light to the camera sensor.

According to a further aspect of the present invention there is provided a method of operating a time of flight camera substantially as described above wherein the data set is processed by performing a spectral analysis to identify frequency values indicative of range information for objects reflecting light to the camera sensor, and frequency values falling within a noise shift frequency band are ignored.

According to yet another aspect of the present invention there is provided a method of operating a time of flight camera substantially as described above wherein the offset phase value at least in part determines the size of the noise shift frequency band.

According to a further aspect of the present invention there is provided a time of flight camera which includes a signal generator configured to generate a source modulation signal and to modify the frequency of the source modulation signal by adding or subtracting at least one multiple of an offset frequency value and modifying the phase of the source modulation signal by adding or subtracting at least one multiple of an offset phase value, a camera light source configured to transmitting light modulated by a modulation signal generated by the signal generator a camera sensor configured to capture time of flight camera data from received reflected light, a processor configured to compile a data set from captured time of flight data frames and to process said data set to determine range information for objects reflecting light to the camera sensor.

According to yet another aspect of the invention there is provided a computer readable medium embodying a program of computer executable instructions arranged to operate a time of flight camera, the program of instructions including: at least one instruction to generate a source modulation signal, at least one instruction to transmit light modulated by the source modulation signal from a camera light source and to capture a time of flight camera data frame from a camera sensor illuminated with reflected light modulated by the source modulation signal, at least one instruction to modify the frequency of the source modulation signal by adding or subtracting a multiple of an offset frequency value and to modify the phase of the source modulation signal by adding or subtracting a multiple of an offset phase value to generate a stepped modulation signal, at least one instruction to transmit light modulated by the stepped modulation signal from the camera light source and to capture a time of flight camera data frame from the camera sensor illuminated with reflected light modulated by the stepped modulation signal, at least one instruction to modify the frequency of the stepped modulation signal by adding or subtracting a multiple of the offset frequency value and modify the phase of the stepped modulation signal by adding or subtracting a multiple of the offset phase value to generate an updated stepped modulation signal, at least one instruction to transmit light modulated by the updated stepped modulation signal from the camera light source and capture a time of flight camera data frame from the camera sensor illuminated with reflected light modulated by the updated stepped modulation signal, at least one instruction to generate one or more further updated stepped modulation signals and to capture one or more further time of flight camera data frames using said updated modulation signals to provide a time of flight camera data set, and at least one instruction to process the data set to determine range information for objects reflecting light to the camera sensor.

Various aspects of the present invention can provide a time of flight camera, a method of operating such a camera, and/or a program of computer executable instructions configured to operate a time of flight camera. Reference throughout this specification in general is predominantly made to the invention providing a method of operating a time of flight camera, while those skilled in the art should appreciate that this should in no way be seen as limiting. In various aspects the invention may be embodied by a time of flight camera incorporating a signal generator, and at least one camera light source, camera sensor and processor - this processor or processors preferably programmed with executable instructions which implement the method of operation discussed below.

Those skilled in the art should also appreciate that the components or hardware employed to form this time of flight camera may be drawn from or provided by existing prior art time of flight cameras. Such existing cameras may be readily modified or configured to generate and modify modulation signals, to transmit modulated light and also to capture and process camera data frames using forms of existing camera signal generators, light sources, sensors and processors.

The present invention provides for the capture and processing of a plurality of time of flight camera data frames which are compiled together to define a time of flight camera data set. Those skilled in the art will appreciate that this approach is a common practice with time of flight cameras, which capture multiple measurement data frames processed together as a single data set to provide a single image encoding range information. Those skilled in the art will also appreciate that the number of data frames to be compiled into a single data set may also vary in different embodiments of the invention, as referenced further below.

Each camera data frame is captured with the use of a modulation signal employed by a camera light source. This modulation signal is used by the light source to modulate light transmitted towards objects which are to have their range to the camera measured. The modulated light is then reflected from these objects towards and onto the time of flight camera sensor.

The present invention utilises a different modulation signal in respect of each captured data frame used to compile an entire camera data set.

One data frame may be captured using a source modulation signal which can set a baseline or initial signal. For each subsequently captured data frame the modulation signal used may be formed by a modified version of the source modulation signal and/or the modulation signal used to generate the previously captured data frame. For example, in some embodiments the first data frame may be captured using light modulated by the source modulation signal. A second data frame may then be captured using a stepped modulation signal, being a modified version of the source modulation signal. A third data frame may then be captured using yet another modulation signal, preferably being an updated form of the stepped modulation signal, which itself is a modified version of the original source modulation signal. Those skilled in the art appreciate that updated stepped modulation signals may be generated for the required number of frames used to form a data set processed by the time of flight camera.

Preferably a previously used modulation signal may be modified to capture a new data frame by modifying the frequency of the signal by an offset frequency value and by modifying the phase of the signal by an offset phase value.

In some embodiments both a single multiple of the offset frequency value and offset phase value may be added each time a modulation signal is to be modified. Alternatively both a single multiple of the offset frequency value and offset phase value may be subtracted each time a modulation signal is to be modified. In such embodiments this approach will therefore linearly increase or decrease modulation signal frequency and phase as camera data frames are captured for a single camera data set.

In such embodiments the invention provided may therefore function in a similar manner to an existing stepped frequency continuous wave time of flight camera. However instead of only linear stepped changes being applied to just the frequency of the modulation signal used, linear stepped changes are also applied to the phase of this modulation signal.

However, in other alternative embodiments different multiples of the offset frequency value and offset phase value may be added or subtracted between each consecutive use of a modulation frequency. In such embodiments the multiple of the offset frequency and phase value being added or subtracted is selected to maximise deviation in the modulation frequency used to capture consecutive data frames of a dataset. Preferably the same multiple is used to determine both the overall offset in frequency and phase, and the same operation of either addition or subtraction is used to modify the frequency and phase of a modulation signal.

For example in an embodiment where a dataset is to be composed from 'h' data frames, a data frame may be captured initially with a source modulation signal, and then 'h' times a step frequency value and phase value may be added or subtracted to or from the source modulation signal to generate a stepped modulation signal which is used to capture the next frame of the dataset. In such embodiments all the remaining frames of the dataset may be captured with different modulation frequencies by the addition or subtraction of multiples of the offset frequency and phase value which is less than 'h'. These frames may therefore be captured one after the other in a sequence which utilises different modulation frequencies with different phases and which maximises the deviation between consecutively used modulation frequencies.

Preferably the captured and compiled camera data set may be processed by performing a spectral analysis to identify frequency values indicative of range information for objects reflecting light on to the camera sensor. For example in some embodiments a Fourier transform may be applied to the camera data frames of the data set with the transformed data providing information in the frequency domain. Existing prior art analysis techniques may be used to isolate or identify particular frequency values which are associated with objects reflecting light to the camera sensor, with identified frequency values correlating with the object's range to the camera.

In embodiments with maximised deviation between consecutively used modulation frequencies this spectral analysis may also be used to identify the presence or absence of motion. In such embodiments additional frequency components will be incorporated into the results of such a spectral analysis when the camera observes a jump in distance values for at one or more one pixels due to an object moving laterally as the frames of the data set are being captured. In such embodiments the presence of anomalous spectral peaks will identify the movement of an object and potentially that the data set may need to be recaptured due to contamination by motion based error or noise.

Those skilled in the art will appreciate that the values of these additional frequency components depend on when the motion occurs during the frame acquisition process, and the specific ranges of objects within the view and range of the time of flight camera.

The presence of additional frequency components which are induced by motion may be detected in a preferred embodiment using a temporal or spatial filter to determine a sudden change in spectral content compared to the previous acquired data frame, or neighbouring pixels within the same data frame. In yet other embodiments this motion detection process may be completed by assessing the number of objects detected at different ranges within a particular frame, and identifying that motion has occurred if the number of objects identified exceeds a threshold number and therefore is highly likely to be the result of motion of objects in front of the time of flight camera.

In a preferred embodiment the frequency values falling within a noise shift frequency band may be ignored during the process used to identify frequency values indicative of range information. The start and end values of the noise shift frequency band may vary in different embodiments depending on how a data set is captured. As frequency values correlate to distance values this noise shit frequency band can also be represented by a distance which extends from the location of the camera.

The extent or size of the noise shift frequency band relates at least in part to the offset phase value employed to modify or step change the modulation signal used in the capture of each successive data frame of the data set being processed. The application of the offset phase value results in a global shift in the frequency values related to light reflected from objects on to the camera sensor. Conversely a variety of signal noise or error sources embedded in the camera data frames do not experience this global offset shift in frequency values. The application of the offset phase value to the modulation signals used translates object reflection generated frequency measurements away from noise or error generated frequencies. These error sources normally present in prior art step frequency continuous wave time of flight cameras as false positive range values for objects close to the camera. The present invention may therefore be utilised to mitigate, reduce or potentially remove such sources of error in the range information it provides.

For example, in one preferred embodiment the extent of the noise shift frequency band represented as distance in front of the time of flight camera is determined from the expression: cA§

4tt /

Where c is the speed of light, DQ is the offset phase value, and &f is the offset frequency value.

In some embodiments the number of data frames used to form a data set may also be chosen to set the size of the noise shift frequency band in combination with the offset phase value. Those skilled in the art will however appreciate that the specific application in which the time of flight camera provided is used can dictate how many data frames are required for each data set, as can limitations related to the performance of the camera sensor or processor provided. For example, in one additional embodiment the extent of the noise shift frequency band represented as distance in front of the time of flight camera is determined from the expression:

Where c is the speed of light, DQ is the offset phase value, D/ is the offset frequency value, and N is the number of data frames in the sequence.

The present invention may provide potential advantages over prior art. In particular the present invention may provide improvements in relation to prior art step frequency continuous wave time of flight camera systems to mitigate or reduce signal noise errors experienced by these devices. The present invention can facilitate the definition of a noise shift frequency band within the results obtained by a time of flight camera, allowing the results falling within this band is to be ignored, thereby isolating valid object reflection measurements from sources of error.

In various embodiments the present invention may also provide more accurate results when compared with equivalent AMCW and SFCW time of flight camera systems. As the effective frequency of the light intensity measured at the camera sensor is increased when compared with prior art time of flight cameras, this characteristic of the invention increases the number of observable zero crossing points for the measured return signal - thereby allowing existing signal processing techniques to more accurately evaluate the measured return signal and therefore the range measurements derived from it.

Brief description of the drawings

Additional and further aspects of the present invention will be apparent to the reader from the following description of embodiments, given in by way of example only, with reference to the accompanying drawings in which:

• figure 1 shows a block schematic diagram of the components of the time of flight camera provided in accordance with one embodiment of the invention, • figure 2 shows a flowchart of a program of computer executable instructions arranged to operate the time of flight camera of figure 1, and

• figure 3 shows a plot of single pixel raw amplitude values recorded during the capture of a sequence of camera data frames using a prior art step frequency continuous wave camera and a camera provided in accordance with the embodiment of figures 1 and 2,

• figure 4 shows a plot of the spectral transformation of the camera data frames illustrated with respect to figure 2 after the performance of an offset calibration shift, and

• figures 5a, 5b and 5c show plots of modulation signal frequency with a data frame index, modulation signal phase offset with the same data index, and object distance with modulation frequency for an exemplary camera data set captured in accordance with one embodiment, and

• figures 6a, 6b and 6c show plots of modulation signal frequency with a data frame index, modulation signal phase offset with the same data index, and object distance with modulation frequency for an exemplary camera data set captured in accordance with a further embodiment to that referenced with respect to figures 5a through 5c.

Further aspects of the invention will become apparent from the following description of the invention which is given by way of example only of particular embodiments.

Best modes for carrying out the invention

Figure 1 shows a block schematic diagram of the components of the time of flight camera 1 provided in accordance with one embodiment of the invention. The camera 1 incorporates the same components as those utilised with a prior art step frequency continuous wave time of flight camera including a signal generating oscillator 2, light source 3, light sensor 4 and processor 5. The processor 5 is programmed with a set of executable instructions which control the operation of each of the remaining components, as described further with respect to figure 2.

Figure 2 shows the first step A of this operational method where the signal oscillator generates a source modulation signal. Step B is then executed with the light source transmitting light modulated by the source modulation signal and the light sensor capturing a camera data frame.

At step C instructions are executed to modify the source modulation signal with the addition of a frequency offset value and a phase offset value to provide a stepped modulation signal.

Step D is then executed with the light source transmitting light modulated by the modulation signal generated at step C with the light sensor capturing a further camera data frame.

At step E an assessment is made of the number of data frames captured so far when compared with the number of data frames required for a complete time of flight camera data set. If the data set is incomplete the process returns to step C where the last modulation signal used is modified with the addition of the frequency offset value and the phase offset value.

Once a complete data set has been captured step F is completed to perform a spectral transformation on the captured data frames. In the embodiment shown the spectral transformation is performed using a Fourier transform.

Next at step G a calibration operation is completed on the transformed spectral data set to map each frequency in the data set with a corresponding distance to the time of flight camera. In the embodiment shown frequencies in the noise shift frequency band will have a negative distance to the time of flight camera.

At step FI any frequency values related to negative distances which are present in the spectral data set are discarded from further consideration. Lastly at step I the remaining values present in the data set are analysed to identify particular frequency and corresponding distance values which are associated with objects reflecting light to the camera sensor, with identified distance values correlating with the object's range to the camera.

Figure 3 shows a plot of single pixel raw amplitude values recorded during the capture of a sequence of camera data frames. The two plots shown were provided using a prior art step frequency continuous wave camera (plotted as '*') and a camera provided in accordance with the embodiment of figures 1 and 2 (plotted as 'c'). These figures clearly contrast how the phase offset value applied to the modulation signal results in a higher frequency trace than with a prior art SFCW camera.

Figure 4 shows a plot of the spectral transformation of the camera data frames illustrated with respect to figure 3. In the embodiment shown the method of the invention results in an effective subtraction of approximately six meters of the equivalent frequency results, with negative distance values present in this spectra corresponding to frequencies present in the noise shift frequency band.

Considering the prior art SFCW cameras results are plotted as '*', it is clear that the large peak shown at around zero metres is a mixture of results from a reflected object and general camera noise and error. The top of this main peak is offset from the true distance indicator shown.

The results provided in accordance with the above referenced embodiment of the invention are plotted as the 'c' data points, where the noise isolated by the invention sits in the noise shift frequency band running from the equivalent range values of -6 to 0 meters. Consequently, the invention is able to separate these noise-based results from object reflections, resulting in the major 'c' plotted peak shown at approximately 0.5m. This peak has a lower amplitude than the prior art SFCW peak but more accurately reflects the true distance to the object in front of the camera at approximately 0.5 m.

Figures 5a, 5b and 5c show plots of modulation signal frequency with a data frame index, modulation signal phase offset with the same data index, and object distance with modulation frequency for an exemplary camera data set captured in accordance with one embodiment.

Plot 5a identifies the modulation frequency value used to capture a sequence of 11 data frames making up the camera dataset. The frame acquisition sequence starts with the capture of data frame 1 and ends with the capture of data frame 11. Similarly plot 5b identifies the modulation signal phase offset applied to the modulation signal as data frames 1 through 11 are captured.

In the embodiment shown an offset frequency value of 1 MHz is used with an offset phase valve of 30 degrees. Frame 1 is captured using a source modulation signal of 1MHz which has no phase offset applied. Frame 2 is captured with a stepped modulation signal formed by adding a single multiple of the offset frequency value to the frequency of the source modulation signal and by adding a single multiple of the offset phase value to the source modulation signal. Each successive frame is captured using an updated stepped modulation signal which adds a single multiple of the offset frequency value to the frequency of the previously used modulation signal and by adding a single multiple of the offset phase value to the previously used modulation signal.

An object within the view of the associated time of flight camera initially sits a distance of 2m from the camera and moves to a distance of 6m between the acquisition of data frames 5 and 6. The change in distance between the camera and this object is illustrated in the third plot shown when the object is identified at 2m from the camera while the first five frames are captured, and then 6m from the camera when frame number 6 through 11 are captured.

Figures 6a, 6b and 6c show plots of modulation signal frequency with a data frame index, modulation signal phase offset with the same data index and object distance with modulation frequency for an exemplary camera data set captured in accordance with a further embodiment.

Again plot 6a identifies the modulation frequency value used to capture a sequence of 11 data frames making up the camera dataset. Similarly plot 6b identifies the modulation signal phase offset applied to the modulation signal as data frames 1 through 11 are captured.

In the embodiment shown an offset frequency value of 1 MHz is used with an offset phase valve of 30 degrees. Frame 1 is captured using a source modulation signal of 1MHz which has no phase offset applied. The modulation signals used to capture the remaining frames of the data set are ordered in this embodiment to maximise deviation between the modulation signals used to capture consecutive data frames.

Frame 2 is captured with a stepped modulation signal formed by adding nine times the offset frequency value to the frequency of the source modulation signal. Similarly, and as shown by plot 6b, the same stepped modulation incorporates a phase offset of nine times the offset phase value added to the source modulation signal.

Frame 3 is captured with an updated stepped modulation signal formed by subtracting seven times the offset frequency value from the frequency of the stepped modulation signal used in the capture of frame 2. Seven times the offset phase value is also subtracted from the stepped modulation signal used in the capture of frame 2 to provide the updated stepped modulation signal used to capture frame 3.

Successive updated modulation signals are used to capture frames 4 through 11 where;

• 5 times the offset frequency value and offset phase value is added to the prior updated stepped modulation signal to capture frame 4,

• 3 times the offset frequency value and offset phase value is subtracted from the prior updated stepped modulation signal to capture frame 5,

• 1 times the offset frequency value and offset phase value is added to the prior updated stepped modulation signal to capture frame 6,

• 1 times the offset frequency value and offset phase value is added to the prior updated stepped modulation signal to capture frame 7,

• 3 times the offset frequency value and offset phase value is subtracted from the prior updated stepped modulation signal to capture frame 8,

• 5 times the offset frequency value and offset phase value is added to the prior updated stepped modulation signal to capture frame 9,

• 7 times the offset frequency value and offset phase value is subtracted from the prior updated stepped modulation signal to capture frame 10,

• 9 times the offset frequency value and offset phase value is added to the prior updated stepped modulation signal to capture frame 11 Plot 6c again identifies an object within the view of the associated time of flight camera initially sitting at a distance of 2m from the camera and moving to a distance of 6m between the acquisition of data frames 5 and 6. As can be seen from this plot this object is illuminated using modulation signals of 1, 3, 5, 8 and 10 MHz when at the 2m position, and modulation signals of 2, 4, 6, 7, 9 and 11 when at the 6m position.

In the preceding description and the following claims the word "comprise" or equivalent variations thereof is used in an inclusive sense to specify the presence of the stated feature or features. This term does not preclude the presence or addition of further features in various embodiments.

It is to be understood that the present invention is not limited to the embodiments described herein and further and additional embodiments within the spirit and scope of the invention will be apparent to the skilled reader from the examples illustrated with reference to the drawings. In particular, the invention may reside in any combination of features described herein, or may reside in alternative embodiments or combinations of these features with known equivalents to given features. Modifications and variations of the example embodiments of the invention discussed above will be apparent to those skilled in the art and may be made without departure of the scope of the invention as defined in the appended claims.