Technical Field

[0001] The embodiments disclosed herein relate to image sensing and processing technology, and, in particular to systems and methods for all-analog image processing.

Introduction

[0002] A typical complementary metal-oxide semiconductor (CMOS) image sensor array 100 is shown in FIG 1 . Each pixel in the sensor array 100 is constructed of a photodiode which integrates charge onto a capacitor to generate a voltage on the capacitor proportional to the intensity of the sensed light. Each pixel’s voltage is selected by a row and column addressing scheme to present each individual pixel value to the output through an output amplifier. In typical systems today, the output voltage is then digitized by an analog-to-digital converter (ADC) to provide a binary representation of each pixel for further processing by an image signal processing function which may include artificial intelligence processing.

[0003] The typical CMOS sensor array method consumes a significant amount of power in the ADC function as well as in the subsequent data transport and transmission. In systems which require extremely low power operation, for example, battery powered camera functions for image sensing and for performing always-on image processing, the power overhead of the ADC function, data transmission and the subsequent digital signal processing in the image signal processor can be prohibitive. In addition, CMOS imagers can be utilized in systems that experience wide temperature excursions and for high temperatures that CMOS imager’s performance may degrade. In such cases where high temperatures can be experienced it is critical that the CMOS imager and associated processing circuitry consume minimal power and thus generate minimal heat to minimize further raising the ambient temperature of the CMOS sensor and further degrading performance.

[0004] It is highly advantageous to improve the energy efficiency of such imaging systems by eliminating the digital processing by utilizing analog processing for critical, always-on functions and thermally sensitive applications. These functions could include person detection, face detection and specific object detection and automotive applications where the camera module is not easily cooled and can experience high ambient temperatures. Additionally, an all analog neural network could be used to improve the image quality of the CMOS imager. Accordingly, there is a need for a low-power consuming solution which directly processes the analog voltages from the image sensor and utilizes a neural network computational system to perform the always-on image processing functions.

Summary

[0005] Systems and methods for all analog image processing are described. The systems are configured to convert a pixel output signal to a time pulse without analog to digital signal conversion. The output time pulse is proportional to the pixel voltage and may be processed by an all-analog neural network to perform functions such as object detection, object classification, image quality enhancement and always-on image processing with low power consumption.

[0006] According to some embodiments, there is an analog image processing system. The system comprises a pixel array comprising a plurality of pixels for sensing light and a buffer for sampling a pixel voltage output by each pixel in the pixel array. A switch passes the pixel voltage to a capacitor, wherein the pixel voltage is applied to the capacitor when the switch is on, and wherein a discharge current is applied to the capacitor when the switch is off. The system includes a comparator configured to sense the pixel voltage on the capacitor as the capacitor is discharged and generate output time pulses proportional to the pixel voltage.

[0007] According to other embodiments, there is an analog image processing system comprising a pixel array comprising a plurality of pixels, each pixel having a photodiode for sensing light, a capacitor for storing charge generated from the photo-diode and a switch for enabling a discharge from the capacitor. A comparator is configured to sense a diode voltage on the capacitor as the capacitor is discharged and generate an output time pulse proportional to the diode voltage. [0008] The systems disclosed herein may further comprise an analog neural network for processing the output time pulses, the analog neural network having two or more layers of neurons and synapses comprising programmable weight-storing charge- trapped transistors. The systems may further comprise a current source for supplying the discharge current, the current source calibrated to a reference discharge current by a charge-trapped transistor programming a threshold voltage shift into the current source thereby creating a shift in the discharge current.

[0009] Other aspects and features will become apparent, to those ordinarily skilled in the art, upon review of the following description of some exemplary embodiments.

Brief Description of the Drawings

[0010] The drawings included herewith are for illustrating various examples of articles, methods, and apparatuses of the present specification. In the drawings:

[0011] FIG. 1 is a typical CMOS image sensor array;

[0012] FIG. 2 is a diagram of an all-analog signal path for image processing, according to an embodiment;

[0013] FIG. 3A is a diagram of an analog voltage to pulse conversion image processing system, according to an embodiment;

[0014] FIG. 3B is a timing diagram of an exemplary voltage to pulse conversion in the pulse conversion circuit of FIG. 3A, according to an embodiment;

[0015] FIG. 4 is a diagram of an analog voltage to pulse conversion image processing system, according to another embodiment; and

[0016] FIG. 5 is a flow diagram of an analog image processing method, according to an embodiment.

Detailed Description

[0017] Various apparatuses or processes will be described below to provide an example of each claimed embodiment. No embodiment described below limits any claimed embodiment and any claimed embodiment may cover processes or apparatuses that differ from those described below. The claimed embodiments are not limited to apparatuses or processes having all of the features of any one apparatus or process described below or to features common to multiple or all of the apparatuses described below.

[0018] A description of an embodiment with several components in communication with each other does not imply that all such components are required. On the contrary a variety of optional components are described to illustrate the wide variety of possible embodiments of the present invention.

[0019] Further, although process steps, method steps, algorithms or the like may be described (in the disclosure and I or in the claims) in a sequential order, such processes, methods and algorithms may be configured to work in alternate orders. In other words, any sequence or order of steps that may be described does not necessarily indicate a requirement that the steps be performed in that order. The steps of processes described herein may be performed in any order that is practical. Further, some steps may be performed simultaneously.

[0020] When a single device or article is described herein, it will be readily apparent that more than one device I article (whether or not they cooperate) may be used in place of a single device I article. Similarly, where more than one device or article is described herein (whether or not they cooperate), it will be readily apparent that a single device I article may be used in place of the more than one device or article.

[0021] Referring to FIG. 2, shown therein is a diagram of an all-analog signal pathway 200 for image processing, according to an embodiment. The all-analog signal pathway 200 may provide for low energy consumption in always-on image processing applications in battery operated devices and thermally sensitive applications. The pathway 200 may provide a method for interfacing directly to a CMOS imager with the data converter replaced by a voltage to pulse width conversion.

[0022] Considering the pixel array 100 in FIG. 1 and representing the array as a single block 202 in FIG. 2, the output of the pixel array 202 at any point in time is the output voltage of the pixel for whom the row and column have been selected. The output voltage looks like a discrete time signal which changes with each row and column selection which is often referred to as the “pixel clock rate”. The pixel clock sets the rate at which pixel data is output from the array and is related to the number of pixels in the array and the video frame rate desired (see FIG. 3A)

[0023] Similar to the typical CMOS image sensor array, in the pathway 200, the discrete time voltage output from the pixel is buffered by the voltage buffer 204 in the pixel array 202. The output of the pixel voltage buffer 204 is then converted to a time pulse signal by a voltage to pulse width conversion function 206, in which the duration of the pulse in time is proportional to the pixel voltage. For example, if a 1 ps pulse width represents the maximum pixel voltage, for example 1 V, then a 500 mV voltage would generate a 500 ns pulse width, being the full scale value. This has similarities in some ways to analog-to-digital conversion but with significant, important differences.

[0024] The voltage to pulse conversion 206 performs a quantization of the input voltage in time but does not digitize the voltage into a numerical representation, such as a binary representation. This maintains the full analog dynamic range of the signal and simply transforms the signal from voltage to time. The voltage to pulse conversion function can be implemented in many different technologies and in CMOS technology can achieve low power consumption on the order of 10s of nW.

[0025] The all-analog signal pathway 200 includes a neural network processing technique 208 which utilizes time pulses to represent input signals to the neural network and internally using time pulses to represent neural network layer activations, such as described in U.S. Patent Publication No. 2022/0374698, the entirety of which is incorporated by reference herein. The neural network processing technique 208 may be implemented by an analog fully connected neural network (FCNN), an analog convolutional neural networks (CNN), or an analog recursive neural network (RNN) having two or more layers of neurons and synapses comprised of programmable weightstoring charge-trapped transistors (CTTs). Using such a neural network, it is possible to implement many types of analog image processing functions such as object detection, object classification and image quality enhancement, using minimum power. The concept can be generalized and applied to applications beyond CMOS imager-based solutions, in which analog signals carry the important information and a maximally energy efficient neural network or other analog processing solution is required. [0026] Figure 3A shows an exemplary analog voltage to pulse conversion image processing system 300, according to an embodiment. The pulse conversion circuit 300 includes the pixel array 100 and the voltage buffer of FIG. 1 , represented by the pixel array 302 and the voltage buffer 304, respectively. The output voltage of the pixel selected at the pixel clock rate (specifically selected by the row and column select in FIG. 1 ) within the pixel array 302 is buffered by the voltage buffer from FIG. 1 , noted as “pixel voltage” in FIG. 3A, and presented to the sample and hold function shown in FIG. 3A comprised of capacitor “C” and sample switch 308. A comparator 306 is configured to sense the pixel voltage on the capacitor, C, as the capacitor is discharged and generate output time pulses proportional to the pixel voltage.

[0027] When the sample switch 308 is turned on, the pixel voltage is applied to capacitor C. When the sample switch 308 is turned off, the applied pixel voltage is held on the capacitor C until the next time the sample switch 308 is turned on. Once the sample switch 308 is turned off, the discharging current, Idis, is applied to the capacitor, C, and at the same time, the comparator 306 is enabled which causes its output to go high generating the rising edge of the output pulse. As the capacitor, C, discharges, the voltage on the capacitor ramps down until it equals the externally provided or internally generated voltage, Vref, at which point in time the comparator 306 output will go low generating the falling edge of the output pulse. The output pulse remains high during this process until the capacitor voltage is discharged to a level equaling Vref. The amount of time that the output pulse voltage remains high is proportional to the sampled voltage from the pixel voltage buffer 304.

[0028] The sample switch 308 in Figure 3A can be comprised of any type of switch available in the implementation technology. For example, in a CMOS application the switch 308 can be constructed from an N-type metal oxide semiconductor (NMOS) or P- type metal oxide semiconductor (PMOS) transistor or a CMOS switch comprised of both NMOS and PMOS transistors. The discharge current, Idis, can be constructed of any type of current source, such as an NMOS current source available in CMOS technologies. Control signals (not shown in Figure 3A) are implemented using standard logic techniques to enable and disable the discharge current, Idis, and the comparator 306 output during the sampling time. [0029] Figure 3B shows timing diagrams 320, 322, 324, 326, 328 of an exemplary voltage to pulse conversion in the pulse conversion circuit of FIG. 3A. The topmost diagram shows the pixel voltage 320, which is the buffered voltage of the pixel selected within the pixel array. Note that the pixel rate is the period between pixel voltage changes, and is typically equivalent to the sample switch control signal 322. When the sample switch control signal 322 is high, the switch 308 is closed and the output of the pixel voltage buffer 304 is connected to capacitor, C, and the comparator 306 input. During this time both the comparator 306 and Idis 326 are disabled through control logic not shown in the diagram.

[0030] When the sample switch control signal 322 goes low, the sample switch 308 is opened and the capacitor, C, holds the sampled charge. Simultaneously, the Idis 326 and comparator 306 are enabled causing the capacitor, C, to discharge at a controlled rate set by Idis 326 and the comparator output 328 will go high. As the capacitor, C, discharges, the voltage 324 will drop in a linear ramp approaching a voltage equal to the reference voltage of the comparator, Vref. Once the voltage on capacitor, C, reaches the reference voltage of the comparator 306, the comparator output 328 will switch from high to low and the control logic (not shown) will disable the Idis and the comparator 306. The resultant output 328 time pulse width, Tpuise, is thus proportional to the sampled pixel voltage 324. The output time pulses may be processed by a processor or an analog neural network configured for image processing.

[0031] There are many possible circuit implementations for the discharge current and the comparator 306 of Figure 3A. In a preferred embodiment, the discharge current and the comparator 306 are similar structures to those presented in U.S. Patent Publication No. 2022/0374698. Specifically, the discharge current source may comprise a standard NMOS transistor, a NMOS charge-trapped transistor, or a combination thereof, which can be calibrated using a charge-trapped transistor (CTT) device and programming a threshold voltage shift into the NMOS current source creating a shift in the drain current and can be matched to a reference discharge current available in the overall system. [0032] Figure 4 shows an analog voltage to pulse conversion image processing system 400, according to another embodiment. The system 400 is similar to the system 300 shown in FIG. 3, however, the pixel voltage is output from a photo-diode 402 in a pixel array and the voltage buffer 304 is omitted. Such a system 400 may be advantageous for small image sensors with low pixel count, charge-coupled devices, or contact image sensor applications where there is no requirement for bus routing and buffering to get the pixel voltage out.

[0033] In the system 400, the pixel voltage is not sampled and the capacitor, “C,” is directly discharged by the discharge current, Idis. It should be noted that the capacitor, “C,” may be integral to the photo-diode 402 or may be a separate capacitor for storing charge from a plurality of photo-diodes in the pixel array. When the switch 408 is turned on, a comparator 406 configured to sense the diode voltage on the capacitor, outputs a time pulse directly as the capacitor, C, discharges. The output time pulse may be processed by a processor or an analog neural network configured for image processing.

[0034] Figure 5 shows a flow diagram of an analog image processing method 500, according to an embodiment. The method 500 may be implemented by the analog image processing system 300 shown in FIG. 3A. The elements from FIG. 3A are indicated in parenthesis for reference.

[0035] At 502, a pixel in a pixel array (302) senses light (i.e. , photons).

[0036] At 504, optionally, a comparator (306) is disabled. Generally, the comparator (306) will only be disabled if was enabled (i.e., Act 512 is performed) in a prior performance of the method 500. If performed, Act 504 is performed concurrent to Act 506.

[0037] At 506, a pixel voltage from the pixel is sampled by a voltage buffer (304).

[0038] At 508, the pixel voltage is applied to a capacitor (C).

[0039] At 510, a discharge current is applied to the capacitor (C).

[0040] At 512, optionally, the comparator (306) is enabled. Generally, the comparator (306) will only be enabled if was disabled (i.e., Act 504 is performed). That is, according to some embodiments, the comparator may be continuously enabled. If performed, Act 512 is performed concurrent to Act 510. [0041] At 514, the comparator (306) senses the pixel voltage on the capacitor as the capacitor is discharged.

[0042] At 516, the comparator (306) outputs time pulses proportional to the pixel voltage.

[0043] At 518, according to some embodiments, the output time pulses are processed by an analog neural network configured for image processing.

[0044] While the above description provides examples of one or more apparatus, methods, or systems, it will be appreciated that other apparatus, methods, or systems may be within the scope of the claims as interpreted by one of skill in the art.