FOR

HYBRID CONTINUOUS ZOOM LENS SYSTEM AND METHOD

INVENTORS

Yung Yuan KAO

Koon Lin CHEO

RELATED APPLICATIONS

This application claims the benefit of the following patent applications:

U.S. provisional Patent Application S.N. 62/023223, filed 1 1 July 2014;

CIP of U.S. Patent Application S.N. 14/708,163, filed 8 May 2015, which in turn is a 371 and CIP of International Application PCT/IB2013/002905, which in turn claims the benefit of U.S. provisional Patent Application S.N. 61/874,333, filed 5 September 2013, as well as U.S. provisional Patent Application S.N. 61/925,215, filed 61/724,221 filed 8 November 2012;

CIP of International Patent Application PCT/IB2015/000409, filed 8 January 2015, which in turn claims the benefit of U.S. provisional Patent Application SN

61/925,215, filed 8 January 2014.

FIELD OF THE INVENTION

The present invention relates generally to miniature lens systems such as used with cameras in handheld devices and the like, and more particularly relates to miniaturized lens systems having optical zoom at two or more optical power increments, together with software enhancements between the optical increments to provide to the user the impression of substantially continuous optical zoom throughout a range of optical powers.

BACKGROUND OF THE INVENTION

Modern handheld devices such as smartphones typically include a miniaturized camera. In choosing a new smartphone, the quality of the camera has come to be an important factor to many consumers, since most other features are substantially fungible across the best-known platforms.

Smartphone manufacturers have expended a great deal of effort to improve the performance of the cameras in their handheld devices, generally with the goal of providing quality equivalent to at least a traditional point-and-shoot digital camera, but in the form factor of a smartphone. Unlike their traditional counterparts, miniaturized digital cameras must meet dimensional requirements associated with the limited volume available in smartphones and similar devices. In many such devices, the most significant limitation is z-height, since z-height in smartphones has frequently gotten smaller with each generation of device. Because of these volume limitations, conventional autofocus and zoom mechanisms cannot be implemented in devices such as smartphones.

Over time, image quality in smartphone cameras has been improved by increasing the pixel count of the sensor, as well as improving the associated image processing hardware and software. Nevertheless, and again because of volume limitations, virtually all commercially-available smartphones have used a cropping technique, often referred to as "digital zoom", to simulate the zoom feature found in most traditional digital cameras. Digital zoom suffers from several limitations, most significantly that the quality of the image degrades as the user "zooms in" on the image because the "zoom" is achieved by using only a portion of the sensor to create the image. In contrast, optical zoom uses the optics of the lens system to increase the optical power, such that the magnified image is captured by the entire sensor.

Although traditional optical zoom in DSLR's and other high end cameras is true continuous zoom, optical zoom in many digital cameras is a series of steps, or increments, rather than continuous zoom. Likewise, most if not all camera modules in smartphones and similar handheld devices implement digital zoom as a series of increments or steps, although the number of steps can be large enough that the consumer perceives the operation as substantially continuous zoom.

Some vendors of smartphones have attempted to implement optical zoom as external add-ons to their handheld devices, with the result negatives of greatly increased volume, among other things. Another approach, implemented in the Nokia Lumia 1020, is to use a very large sensor, for example 41 megapixels, in combination with digital zoom such that, during zoom, the effective pixel count of the "zoomed in" image is about 3 megapixels.

As a result, there has been a need for a miniaturized optical zoom design suitable for implementation in a smartphone or similar handheld or otherwise small device.

SUMMARY OF THE INVENTION

Recently, miniaturized lens systems using Alvarez-type lenses have been implemented in lens modules suitable for use in smartphones and the like. Alvarez-type lenses are rotationally asymmetric, and move transversely to the optical axis, whereas traditional, rotationally symmetric lenses typically move along the optical axis. The lateral movement of Alvarez-type lenses allows for implementation of optical zoom in a lens module having a volume and z-height generally comparable to "digital zoom" lens modules used in smartphones.

However, because of distortion and other characteristics of some Alvarez-type lens systems, some implementations provide clear images at specified increments of optical power. For example, and as described in more detail hereinafter, in an embodiment an Alvarez-type lens system may be optimized to provide a clear image at 1 X, 1 .4X, 1 .8X, 2.2X, 2.6X and 3X. In an embodiment, in between those increments software is used to provide to the user the benefit of continuous zoom. In some embodiments, digital zoom can be used. Alternatively, interpolation or other software techniques can be used. In still other embodiments, a combination of digital zoom and software techniques can be used to effect continuous zoom in between the increments of optical power.

Various configurations of lenses can be used to provide the increments of optical power. In general, a miniaturized lens system using Alvarez-type lenses for use in some handheld devices is disclosed in commonly-owned application S.N. 14/246,571 , incorporated herein by reference. Further, more detailed embodiments are discussed in commonly-assigned U.S. Patent Application 61/874,333, filed 5 September 2013, and entitled Miniature Optical Zoom Lens, as well as commonly-assigned U.S. Patent Application S.N. 61/925,215, filed 8 January 2014, and entitled Lens Assemblies and Actuators for Optical Systems and Methods Therefor, both of which are incorporated herein by reference, and attached hereto as Appendices A and B, respectively.

The methodologies and algorithms for distortion correction, interpolation, and related image processing which provide a hybrid continuous zoom lens system to the user, in a form factor suited for implementation in a smartphone without requiring an increase in z-height, can be better appreciated from the following detailed description of the invention, including several alternative embodiments, taken together with the following Figures.

THE FIGURES

Figure 1 depicts an optical system in which the optical path is folded twice and varifocal lenses are located in between the folding optics in accordance with an exemplary embodiment.

Figure 2A depicts an optical system with varifocal lenses located at the window, the optical path being folded before reaching the second varifocal lens and folded again before reaching the complementary metal-oxide semiconductor (CMOS) or similar detector in accordance with an exemplary embodiment.

Figure 2B depicts an optical system with a first pair of varifocal lenses located at the entrance window to the module, with one of the elements of the first pair being integrated into the prism. The ray path is folded twice by use of a second prism and the location of a sensor above the second prism, substantially coplanar with the entrance window.

Figure 3 depicts an optical system comprising a varifocal lens element integrated with a prism element and a CMOS detector placed vertically upright in accordance with an exemplary embodiment.

Figure 4 is a ray diagram for an optical system in accordance with the

embodiment of Figure 3, showing one form of movement of the varifocal lenses.

Figure 5 is a ray diagram for an optical system in accordance with the

embodiment of Figure 3, but showing movement of the varifocal lenses in directions opposite to that shown in Figure 4.

Figure 6A illustrates a pair of varifocal lenses that include planar surfaces in accordance with an exemplary embodiment.

Figure 6B illustrates a pair of varifocal lenses wherein each element comprises an Alvarez type surface on one side, and a freeform surfaces on the other side for aberration and distortion correction.

Figures 7A-7B illustrates a sequence of lens movements that provides

incremental optical power from a pair of varifocal lenses, each of two elements.

Figure 8A illustrates a series of zoom steps where optical zoom is used at a series of increments of optical powers, and interpolation is used between the optical increments to provide substantially continuous zoom.

Figure 8B illustrates in system block diagram form the electronics of a lens model in accordance with an embodiment of the invention.

Figure 9 illustrates the effect of interpolation plus distortion compensation on zoom steps relative to focal length.

Figure 10 plots zoom factor versus distortion level across increments of optical zoom and then digital zoom.

Figure 1 1 plots steps of optical zoom, digital zoom, stepless interpolation, and stepless interpolation with distortion compensation. Figure 12 illustrates a first method for implementing optical zoom with distortion correction plus interpolation in a discrete system having a single increment of optical power.

Figure 13 illustrates a first method for implementing optical zoom with distortion correction plus interpolation in an optical system having multiple power increments.

Figure 14 illustrates an alternative method for implementing optical zoom with distortion correction plus interpolation in a discrete system having a single increment of optical power.

Figure 15 illustrates an alternative method for implementing optical zoom with distortion correction plus interpolation in a discrete system having multiple power increments.

Figure 16 illustrates X and Y axis asymmetric distortion in a system comprising rotationally asymmetric lenses.

Figure 17 illustrates distortion resulting from offset and rotation of optical components during assembly.

Figures 18A-B plot curve fitting to compensate for wide angle X and Y distortion in a first quadrant. Diagrams and flipped with 19 in PDF

Figures 19A-B plot curve fitting to compensate for wide angle X and Y distortion in a first quadrant.

Figures 20A-B plot curve fitting to compensate for telephoto X and Y distortion in a fourth quadrant.

Figures 21 A-B plot curve fitting to compensate for telephoto X and Y distortion in a fourth quadrant.

Figure 22A shows an image with barrel distortion, while Figure 22B shows the same image after distortion compensation in accordance with the invention.

DETAILED DESCRIPTION OF THE INVENTION

In applications with limited space, such as a camera phone, miniaturized security camera, etc., the permissible size of the overall camera module significantly influences the configuration of the optical components comprising the module. In some systems, the thickness of the overall device, such as a smartphone, is the most limiting, and therefore the thickness, or z-height, of the module significantly influences the design of the lens module is such applications. In accordance with various embodiments of the present invention, a variety of approaches can be used for managing volume

constraints. One generalized embodiment is shown in Figure 1 , and comprises a window 100 through which light strikes a light path-bending element, such as a prism 105 or a mirror. The prism 105 bends incoming rays 90 degrees and sends them through two varifocal lenses 1 10 and 1 15, respectively. The varifocal lenses, sometimes referred to as 'freeform', are typically Alvarez or Lohmann-type lenses, are rotationally asymmetric, and, as discussed in greater detail hereinafter, provide variable optical power by moving substantially transversely to the optical axis of the lens system. For convenience, it will be generally understood that the term "varifocal lens" as used herein means two elements which cooperate to form a lens, unless otherwise indicated specifically or by context. In an embodiment, the rays then pass through a base lens 120, which can be fixed or moveable depending upon the embodiment, and is, in at least some embodiments, a rotationally symmetric lens. Following the base lens, the light passes through another prism 125, which bends the optical path another 90 degrees so that the rays impinge on a sensor 130. The sensor 130 can be a CMOS detector or any of the other suitable types of sensors that are well known in the art, and can be of any suitable image size, for example eight, eleven, thirteen, sixteen, or more megapixels, and typically referred to as either 1/3" or ½" . In addition, depending upon the embodiment, the sensor can be flat or curved. In an embodiment, the base lens 120 is integrated with the prism 125. In some embodiments, an aperture 135 is placed between the two varifocal lenses 1 10 and 1 15. Such a configuration offers a small z- height for the lens module, in some embodiments approximately 6 mm or less, but in some instances may provide only a limited field of view and larger f-numbers (i.e., smaller aperture) because the combination of an entry window and a prism does not facilitate either light gathering or spreading.

An alternative approach is shown in Figure 2A, where in some embodiments at least one of the varifocal lenses is positioned in advance of a first prism to increase the FOV. In Figure 2A, light enters the lens system through a window 200, then passes through a first varifocal lens 205, and then a prism 210 which bends the light 90 degrees. As the light exits the prism 210, it passes through an optional aperture 215 and then through a second varifocal lens 220. The light thereafter passes through a bases lens 225, and finally through a second prism 230 until the light impinges on a sensor 235. In an embodiment constructed in accordance with Figure 2, a field of view in the range of approximately 60 degrees to 75 degrees can be achieved, although in some instances the z-height may increase to about 8 mm, and larger still in some embodiments such as those intended for, for example, security cameras. It should be noted that the second prism 230 can be oriented either to bend the light upward or downward, merely as a matter of design choice, but bending the light upward allows the sensor to be essentially co-planar with the first varifocal lens 205, thus reducing z-height as compared to bending the light downward onto a sensor.

A still further alternative embodiment is shown in Figure 2B, where like elements are shown with the same reference numerals as for Figure 2A. Figure 2B again uses a two prism design but, in this embodiment, incorporates into the upper surface of the first prism one of the varifocal lenses of the first pair of varifocal lenses. In addition, the second prism is inverted relative to the design shown in Figure 2A, with the sensor placed above the second prism .

Referring next to Figures 3 and 4, Figure 3 illustrates an alternative approach that eliminates the second prism and shortens the path length between the entry window and the sensor, while Figure 4 is a ray diagram that illustrates the ray paths through the lens system of Figure 3. As with the embodiments of Figures 1 and 2, light passes through a window 300, but then passes through a varifocal lens 305 where one of the rotationally asymmetric elements is the upper, or entry, surface of prism 310. Light exits the rear surface of the prism 310 and passes through aperture 315, then through a second varifocal lens 320, and then through base lens 325. By repositioning the sensor 330 to be substantially orthogonal to the rays exiting the base lens, the second prism of Figures 1 and 2 is eliminated, and light exiting from the base lens 325 impinges directly on the sensor 330. As noted before, in some embodiments the base lens is fixed, while is others the elements are movable either jointly or separately, either for achieving focus or additional optical power or both. In embodiments where at least one element in the base lens is moveable, the base lens element or elements move along the optical axis to change the optical power of the base lens and compensate for any shift in the location of the image plane that occurs when the elements of the pair of varifocal lenses are moved. It will be appreciated by those skilled in the art that, in an embodiment, the primary change in optical power of the lens system occurs through the relative movement of the elements of the varifocal lenses, with the adjustment of the base lens providing a secondary change in optical power. In an embodiment, the overall optical path length of a lens module as shown in Figure 3 is approximately 18 mm with a z-height of about 6 mm, with a field of view of about 60-75 degrees. In some embodiments, for example for 1/3" sensors, the height of the lens module is generally about 1 .2 - 2 times the height of the image sensor , although the relationship of module height to sensor height can vary with sensors of different sizes. Optionally, the first varifocal lens 305 can be formed separately from the prism 310, although this generally results in a greater z-height to accommodate the extra lens element. As a still further alternative, the module height can be further reduced by cutting off a portion of the top and bottom of the lens elements that form the base lens, with only modest adverse impact on relative illumination. In some embodiments, for example those using 1/3" sensors, such an approach can reduce module height to less than six millimeters. An additional feature of the lens system of the present invention is the exceptionally short stroke, where, in an embodiment such as shown in Figure 2B with a 1/3" sensor, the full range of optical power can be achieved with a stroke of approximately two to two-and-a- half millimeters for the Alvarez pairs, and a stroke of one to one-and-a-half millimeters for the base lens group. It should be noted that the optical elements of the base lens group need not move together , or even in the same direction, in some embodiments.

In the embodiment of Figure 4, moving the two elements of the first Alvarez lens, shown at 305 and 310, perpendicular to the optical axis in opposite directions (e.g., one lens element is moved out and the other lens element is moved into the page as shown by the arrow point and feather), a negative optical power is produced. Further, in the exemplary diagram of Figure 4, by moving the two elements 405 and 410 of the second Alvarez lens 320 perpendicular to the optical axis at opposite directions, a positive optical power is achieved. The movement of the lens elements can be controlled using one or more actuators that are coupled to the lens elements, and may for example be of the type described in U.S. Patent Application S.N. 61/925,215, filed 8 January 2014 and commonly assigned. The embodiment of Figure 4 produces a miniature lens system with a small height, which makes this configuration particularly advantageous for implementation in devices with thin form factors, such as a cell phone or tablet.

Figure 5 illustrates a ray diagram for the miniature lens embodiment shown in Figure 3, but, in this case, the relative motion of the varifocal lenses is the opposite of that shown in Figure 4. As with Figure 4, the various optical components in Figure 5 are positioned to provide the desired optical power. Unlike Figure 4, in Figure 5 the element 305 of the first varifocal lens is shown moving inward, while the element 310 is shown moving outward, creating a positive optical power. In contrast, the elements 405 and 410 of second varifocal lens 320 are shown moving outward and inward, respectively, creating a negative optical power.

Referring next to Figures 6A and 6B, two varifocal lenses of the type shown in Figures 1 -5 can be seen in greater detail. In particular, Figure 6A shows varifocal lens 600, with the freeform surfaces of elements 600A and 600B opposing one another. Likewise for varifocal lens 605, the freeform surfaces of lens elements 605A-605B are shown opposing one another . In comparison, Figure 6B shows the varifocal lenses 650 and 655 with the freeform or Alvarez surfaces on the same side of each element 650A- B and 655A-B, respectively. In addition, Figure 6B shows the opposite sides, indicated at 660, of each of the elements shaped to correct for aberrations and distortion. In at least some embodiments, the respective surfaces are typically characterized by a polynomial, with some of the terms of the polynomial defining the curvature that provides the intended optical power, and higher order terms correcting for aberration and distortion. The polynomial defining each surface of each element need not be the same in all embodiments.

As discussed above, optical power of the lens system of the present invention is adjusted by moving the elements of each varifocal lens relative to one another, with one pair effecting a positive optical power and the other effecting a negative optical power. In an embodiment, such as shown in Figures 7A-7B, the elements 700, 705, 750 and 755 of the varifocal lenses are moved incrementally, providing incremental increases (or decreases) in optical power with each increment. In the particular embodiment of Figures 7A-7B, optical power is incremented in steps of 1 X, 1 .4X, 1 .8X, 2.2X, 2.6X and 3X, although other embodiments provide substantially greater ranges of optical power, for example 6X and 10X. In an embodiment, an approximately 2mm relative lateral movement, or stroke, of the varifocal lenses can provide 3X power, although in another embodiment a stroke of approximately 2mm can provide 6x power depending upon the design of the varifocal lenses and the overall lens system. In addition, in embodiments that are not as space-constrained as cellular phones, for example security cameras and the like, a longer stroke is acceptable, for example 5mm or more, and may allow the implementation of, for example, larger sensors and other features. An additional alternative, where space permits, is to provide multiple cameras having different ranges of optical zoom, for example a first camera with a 1x-3x zoom, and a second camera with a 4x-6x zoom. Other combinations are readily possible, given the teachings herein, for example a 1 x camera plus a second camera with a range of 2x-6x, or a third camera having a different range of optical power. As noted above, each of these designs may be optimized for specific increments of optical power.

In any of the aforementioned embodiments, digital interpolation can be used between the optical power increments to provide to the user a smoothly continuous zoomed image. In addition, for digitally zoomed images, where the digital zoom is primarily achieved by cropping of the sensor image, interpolation can be used to eliminate the perception of increments. An example of such smoothing is shown in Figure 8, where a combination of optical zoom, incremental digital zoom, and stepless interpolation is used to provide to the user the impression of smooth, continuous zooming from the minimum to the maximum focal lengths of the lens system. The darker gray area, indicated at 800, represents the step of optical zoom, the intermediate gray area, indicated at 805, indicates two step, or increments, of digital zoom/cropping, and the lightest gray area, indicated at 810, represents the smoothing achieved by interpolation between the steps. Depending upon the implementation, any suitable algorithm can be used, including, as just some examples, polynomial fit, averaging, weighted averaging, small generating kernel, and so on. An embodiment of a system capable of processing the images including applying appropriate algorithms to perform not only digital zoom but also, if desired, correction of aberrations and distortion can be seen in Figure 8B. In particular, an optical system 800 including autofocus and optical zoom is shown in system block diagram form. In particular, a camera module 805 cooperates with a processor module 810 such as integrated into a smart phone, although the camera module of the present invention can also be implemented independently of a smartphone, such as a security camera or other form of image capture device where a small, or miniature, form factor with optical zoom is desirable.

In the camera module, when the user desires to take a picture (user-driven inputs not shown in Figure 8B for convenience), a driver circuit 815 sends currrent to a lens module 820, and, in an embodiment, initially to a focusing actuator 825 to allow the user to see a clear image. The focusing actuator, described in greater detail hereinafter, automatically adjusts the position of a focusing lens group 830 until a clear image is achieved at a sensor 835, using an image signal processor ("ISP") 840 to provide the necessary feedback to the driver circuit 815 to implement any of the suitable autofocus methods known in the art, such as, for example, contrast detection. The autofocus loop can be appreciated from the dashed line in Figure 8B. It will be appreciated that, while the ISP 840 is shown in Figure 8B as within the phone processor, in at least some embodiments the ISP is included within the lens module 805. In some embodiments, particularly those implemented in smartphones, the outputs from the autofocus algorithm existing in the smartphone processor are converted into inputs recognizable by the focusing portion of the driver circuit.

If the user desires to zoom in on the subject, as indicated by a user input (again not shown in Figure 8B for convenience), the driver circuit 815 supplies current to a zoom actuator 845 within the lens module 820. The zoom actuator, described in greater detail hereinafter, moves a zoom lens group 850 through a stroke until the user indicates that the amount of zoom is acceptable. The zoom lens group and the autofocus lens group cooperate to achieve both magnification and image clarity at the sensor. Importantly, as discussed above, for the small form factor of the invention described herein, the zoom actuator moves the zoom lenses in a direction essentially lateral to the optical axis, and, in an embodiment, substantially perpendicular to the optical axis. It will be appreciated that the lateral movement need not be substantially perpendicular in all embodiments as long as suitable magnification and acceptable clarity is achieved.

Once the user is satisfied with both the amount of magnification of the image and its clarity, the user "takes" the picture by causing the ISP 840 and graphics processing unit (GPU) 855 to capture the image from the sensor 835. It will be appreciated that the GPU is typically embedded within a modern smartphone, but, in at least some embodiments of the invention, the processor is maintained within a different type of device such as a security camera, computer system, tablet, etc.

In addition to the zoom and autofocus functions, the ISP and GPU can, either separately or collectively, or in a different processor in or associated with the device, apply various algorithms to provide digital zoom or correction for aberrations or distortion. Just as there are a variety of techniques by which digital zoom can be applied, so there are a variety of approaches for applying corrections for aberration, distortion, or both. In some instances, the distortion correction is applied essentially in real time as the user continues to actuate the zoom function, so that the image displayed on the device screen is continuously corrected. In some devices, the amount of correction needed may be small enough that the quality of the image displayed on the device is sufficiently good that the user experience during the taking of a picture is not impaired even without substantially real-time correction. This can result in instances where the screen on the device is sufficiently low resolution, or in instances where the quality of the optical image simply does not require correction for the size of the image displayed during zooming. However, while the image may be acceptable for purposes of composing a photograph, the image is not acceptable when examined in greater detail, such as is often done by users following the taking of a photograph. In such instances, correction for distortion, aberration, or both, is performed after the image is captured. In either approach, the objective is to provide the user with an image of sufficient quality that the user experience is essentially undistorted continuous zoom. Although the following discussion of exemplary embodiments discusses applying the digital zoom and correction algortims in a particular sequence, it will be appreciated by those skilled in the art that the correction steps can be applied at any convenient stage of the image composition and image capture process as long as image quality appropriate to a good user experience for that particular stage in the process is maintained.

In Alvarez-type lenses, distortion may vary with each optical increment because the lens surfaces are defined by a multi-order polynomial. For example, in an

embodiment, optical power increments or steps of 1 X, 1 .4X, 1 .8X, 2.2X, 2.6X and 3X are available. The distortion characteristics for each step can be, and often are, different, and so the algorithms for correction are also different, as discussed in greater detail hereinafter. The result of stepless interpolation with distortion compensation can be appreciated from Figure 9, which plots zoom step versus equivalent focal length. The gray, cross-hatched area indicated at 900 represents the increments of optical zoom, while the black triangular portions indicated at 905 that span between the optical increments represent the smoothing performed by the combination of stepless interpolation together with distortion correction. The process of transitioning from the algorithm that applies during one step, to the algorithm that applies during the next step can be appreciated from Figure 10, where the distortion correction is shown as a series of steps.

Referring next to Figure 1 1 , which plots zoom step versus equivalent focal length, a method of stepless interpolation combined with distortion correction, for a system having both optical and digital zoom, can be better appreciated. As can be appreciated by comparing Figures 9 and 10 with Figure 1 1 , it can be seen that Figure 1 1 exhibits characteristics of both. More specifically, the plurality of steps of optical zoom can be seen at 1 105, with the black triangles like those of Figure 10 designated at 1 1 10, again representing the combination of stepless interpolation and distortion correction. At longer focal lengths, increments of digital zoom can been seen in the medium gray cross-hatched portion, indicated at 1 1 15, with stepless interpolation indicated at 1 120. It will be appreciated by those skilled in the art that the result is a smoothly continuous zoom lens system. It should be noted that the position of the actuator, and therefore the optical power associated with a given position in the stroke of the actuator, is sensed by one or more sensors 835, as shown in Figure 8B. One suitable sensor is a Hall sensor, with appropriate correction for hysteresis.

It will be appreciated from the foregoing that, as the actuator moves from one increment of optical zoom to the next, with digital zoom in between, it is desirable to manage the transition from digital zoom to optical zoom by smoothing the image. This can be accomplished in several ways, some of which correlate the position of the lenses with the digital zooming function. As an example, in some embodiments, digital zoom operates solely based on the image from the prior optical zoom increment. In such an arrangement, the actuator needs to move the lenses to the next optical increment only in time to match the transition from digital zoom to that next increment of optical zoom. In such an embodiment, the transition from digital zoom to optical zoom can be handled by a transitional algorithm. In an alternative arrangement, however, the short stroke of the lens system allows the actuator to move the lenses to the position of the next optical increment faster than the device provides the zoomed image to the user. In such an embodiment, the digital zoom function can be based not only on the image at the prior optical zoom increment, but also on the image at the next optical zoom increment, permitting a smoother transition from increment to increment. Since the image capture required during the composition phase is transient, it is also possible to store only a relatively small portion of the image. For example, even though a camera in a device may be capable of recording 13 megapixel images, the display of that device may be able to display only 2 MP. In such as instance, it is not necessary to capture more data from the sensor than can be displayed on the device while the user is merely

composing the image, and so the transient image captured, displayed and digitally zoomed during the composition phase can be a comparatively low resolution image that can be stored much more rapidly than the image that the user finally captures once they are done composing the image.

In some embodiments, the quality of the display can be used to permit some variation in the acceptable level of distortion from one optical increment to another. Since typical cell phone displays are less than about 3 MP, and frequently much less, an acceptable level of distortion during the period when the user is composing the image may be two percent at one increment of optical zoom, and, for example, four percent at a different increment of optical zoom, whereas the acceptable level of distortion for the captured image may be around 0 .5%.

In contrast, if, for a particular embodiment, it is desired to reduce distortion down to 0.5% during the composition phase, the correction algorithms are combined and, if appropriate, scaled to yield the desired correction. This permits the user to capture an image having only 0.5% distortion at any point in the actuator stroke, with the

processing already complete and with no jarring jumps at different increments of optical power.

Referring next to Figure 12, an embodiment of a process for combining two different powers of optical zoom (including distortion correction) with digital zoom interpolation can be appreciated. At step 1200, the lens system is at a position of 1 X optical power. For that lens position, assume that two percent distortion exists. At step 1205, a distortion correction algorithm is applied, with the algorithm configured to compensate for that specific two percent distortion. Then, as the user causes the lens system to zoom in, digital interpolation is used at step 1210 to create stepless, or continuous, zoom as perceived by the user. Finally, the user's operation causes the lens system to move to a position of 3X optical power, indicated at step 1215. At this point, digital interpolation is no longer needed, and the image is created by the optical system without interpolation. However, when the lenses are positioned for 2X optical power, assume that four percent distortion occurs. Thus, at step 1220, a distortion correction algorithm, configured to correct that four percent distortion, is applied, yielding a clear image.

The example of Figure 12 is simplistic because, for that embodiment, only 1 X and 3X optical powers can be achieved via the optics. However, the example of Figure 12 illustrates the details by which the continuous zoom shown in Figure 1 1 can be achieved, where, for each step, or increment, of optical power, a distortion correction algorithm is applied, with digital interpolation between the steps. A more complex process, showing multiple steps similar to the steps shown in Figure 1 1 , is shown in Figure 13. For the embodiment of Figure 13, the optical power is stepped from 1 X to 1 .4X to 1 .8X, then 2.2X, 2.6X and finally 3X. In between each step of optical power, digital interpolation is applied to create a smoothly zoomed image for the user. At each step of optical power, a distortion correction algorithm is applied, with each such algorithm configured to match the optical distortion occurring at that optical power. Thus, at step 1300, the optical power is 1 X, and distortion is 2%. Therefore, at step 1305, a 2% correction algorithm is applied to the image. When the user chooses to zoom in, the process advances to step 1310, and digital interpolation is applied to the image until the lens system indexes to the next step. For the embodiment shown, the next step is an optical power of 1 .4X, shown at 1315. In this position, the distortion in the lens system is 3.8%, and so at step 1320 a distortion correction algorithm of 3.8% is applied.

If the user continues to choose to zoom in, digital interpolation is again performed, this time between the optical increments of 1 .4X and 1 .8X, as indicated at 1325. At 1 .8X, the appropriate distortion correction algorithm is applied, analogous to steps 1305 and 1320. Then, between 1 .8X and 2.2X, digital interpolation is again applied. The process repeats for the optical step at 2.6X and then up to 3X, shown at step 1330. At 3X power, the distortion is 3%, and so a 3% correction algorithm is applied at 1335. It will be appreciated that the number of steps, or increments, of optical power is largely arbitrary, and the size and number of the increments can vary with the particular design. It will also be appreciated that, because the stroke between increments is so small, the time period during which interpolation is needed is typically very short. Further, in some embodiments, additional digital zoom can be used after the maximum optical power has been reached.

It will also be appreciated that, for some embodiments, the distortion correction algorithm is not applied until an image is actually captured. Alternatively, and

particularly for lens systems with low distortion at the steps of optical power, distortion correction can be incorporated into the interpolation process that is part of the digital zoom step. Stated differently, the interpolated process is modulated by the distortion correction algorithm. Such an approach is shown in Figures 14 and 15, with Figure 14 illustrating a process similar to that shown in Figure 12. In Figure 14, a lens system provides 1 X and 3X optical powers, shown at 1400 and 1410, and uses digital interpolation in between, shown at 1405. However, no distortion correction is provided at the 1 X setting. Instead, distortion correction is incorporated into the interpolation process during the digital zoom phase and smoothes out the distortion correction between the 1 X and 3X distortion factor. Then, when the lens system finishes moving to the 3X position, the 3X image is captured, shown at step 1415, and the appropriate distortion correction is applied to the captured image at step 1420.

Similarly, Figure 15 shows a multi-step process, similar to Figure 13, but again with the distortion correction algorithm applied during the interpolation process of digital zoom. Thus, at step 1500, the lens system is at 1 X, distortion is at 1 %, and no distortion correction is applied. Then, at 1505, digital zoom interpolation is modulated with distortion correction. At step 1510, the lens system is at the next step, 1 .4X, distortion is at 1 .5%, and, again, no distortion correction is applied. Then, at step 1515, the system again uses digital interpolation, and the interpolation algorithm is modulated by the distortion correction algorithm. The process continues for with optical zoom at 1 .8X, followed by digital interpolation, then 2.2X followed by digital interpolation, then 2.6X followed by digital interpolation, and so on until the maximum optical zoom step is reached, shown as 3X at step 1520 in Figure 15. If the user captures the image at 3X, the image is captured at 1525, and the distortion correction appropriate for the 3X setting, in this example 2.5%, is applied at 1530. As before, once the maximum optical zoom power is reached, additional digital zoom steps can be used. If desired, the distortion correction performed at step 1530 in the above example can instead be integrated into the next digital zoom phase.

The distortion correction algorithms discussed above form an aspect of the invention, although such distortion correction and the associated algorithms are not required in all embodiments. For example, the combination of optical zoom and digital interpolation can be used in lens systems not requiring varifocal lenses. To develop the algorithms appropriate for distortion correction for varifocal lenses, the sources of the distortion have to be identified. Types of distortion in a rotationally symmetric lens system often are characterized as either pincushion or barrel distortion. These types of distortion also occur in varifocal lenses. In addition, because of the complex shapes of varifocal lenses, where the optical surfaces are typically defined by high order polynomials, misalignment of the lens in its mount can also create either offset or rotational distortion. In an embodiment of the invention, and to address each of these potential sources of distortion, a distortion profile of the varifocal lens system is developed by characterizing distortion along both the positive and negative x and y axes. For example, the distortion can have a difference in x and y axis, and the distortion ratio may also be different in the +x (along the movement direction of lens) and -x directions. In addition, the distortion profile is adjusted to permit seamless zoom change between two or more fixed and known optical powers, or magnifications. Referring to Figures 16 and 17, these types of distortion can be better appreciated in the context of a varifocal lens. It will be appreciated that distortion can be described by the same function along the axis about which the lens is symmetrical, but is different along the axis in which the lens is asymmetrical. For example, if the lenses are moving right and left, distortion is a mirror image along the X-axis, but not the Y-axis.

To compensate asymmetric distortions on the X and Y axes of varifocal lens systems, the distorted image is partitioned into four parts, +x, -x, +y and -y, with a unit step function (to determine which quadrant) that can be stated as

where

x is the x-coordinate with distortion

y is the y-coordinate with distortion

x' is the x-coordinate with compensation

is the y-coordinate with compensation

z is a zoom factor; and

u[x], u[-x], u[y] and u[-y] are unit step functions

In addition, compensation functions for different distortion ratios in the +x, -x, +y, and -y quadrants are denoted as f _x i , f _X 2, f _y i , and f _Y 2, respectively.

To compensate for offset distortion, such as shown in diagram (a) of Figure 17, the coordinates can be considered an x-axis and y-axis offset. The compensation algorithm can be depicted as x '(x, z)l f « [x - dx] ^■ f _xl (x, z) + w [-X - dx] ^■ f _x2 (x, z) ]

j '( z)J |« [ - · _YL ( y, z) + u [-y - dy] ^■ f _y2 (y, z)J

(2) where

dx and dy are x-axis are y-axis offsets,

x" is the x-coordinate with compensation considering rotation, and

" is the y-coordinate with compensation considering rotation.

To compensate for system rotation distortion of the type shown in block (b) of Figure 17, in two dimensions, a rotation using matrices of the point {χ', ) to be rotated (orientation from positive x' to ) is written as a vector, then multiplied by a matrix calculated from the angle, Θ:

cos Θ - sin 6*

sin Θ cos Θ

(3) where (x", y") are the coordinates of the point that after rotation, and the formulae also can be seen to be

To provide the necessary distortion compensation, second degree polynomials are substituted to give:

x '(x, z) I

Where

/ χ, βχ, Cx, Dx, Εχ, F _x , A _y , By, Cy , Dy, Ey, Fy are functions of zoom factor.

Substituting 2 ^nd -degree polynomials for functions of zoom factor yields:

The result can be appreciated from Figures 18A-B through 21A-B. Figure 18A shows a plot of fifth-order polynomial wide angle X direction distortion in a first (+X, +Y) quadrant, while Figure 18B shows a plot of fifth-order polynomial wide angle Y direction distortion in the first quadrant. Figure 19A is a plot of fifth-order polynomial wide angle X direction distortion in a fourth (+X,-Y) quadrant, while Figure 19B is a plot of fifth-order polynomial wide angle Y direction distortion in a fourth (+X,-Y) quadrant.

Similarly, Figure 20A is a plot of fifth-order polynomial telephoto (zoom) X direction distortion in a first (+X, +Y) quadrant, and Figure 20B is a plot of fifth-order polynomial telephoto (zoom) Y direction distortion in a first (+X, +Y) quadrant. Figure 21 A is a plot of fifth-order polynomial telephoto (zoom) X direction distortion in a fourth (+X, +Y) quadrant. Figure 21 B is a plot of fifth-order polynomial telephoto (zoom) Y direction distortion in a fourth (+X, +Y) quadrant. As a simple example of the final result, a distorted image is shown in Figure 22A, while a compensated image is shown in Figure 22B.

It will be appreciated by those skilled in the art that the distortion compensation algorithm of the present invention can be based on a number of different mathematical approaches. For example, the curve fitting can be performed based on a polynomial expression, such as a linear polynomial, a quadratic polynomial, a cubic polynomial, or an n ^th order polynomial, such as a fourth, fifth or sixth degree polynomial. Alternatively, the curve fitting can be based on an interpolant approach, such as linear, nearest neighbor, cubic spline, or shape-preserving. Still further, the curve fitting can be based on a rational number approach, where the numerator is any of a group comprising a constant, a linear polynomial, a quadratic polynomial, a cubic polynomial, or a higher order polynomial such as 4 ^th or 5 ^th order; and the denominator can be any of a group comprising a linear polynomial, a quadratic polynomial, a cubic polynomial, or a higher order polynomial such as 4 ^th or 5 ^th order.

Having fully described the details of the present invention, including various alternatives, those skilled in the art will recognize that numerous alternatives and equivalents exist which do not depart from the invention. As a result, the invention is not to be limited to the details described hereinabove, but only by the appended claims.