Field of Invention

[0001] This application claims priority under Article 8 of the Patent Cooperation Treaty (PCT) to U.S. Non-Provisional No. 15/173,306, filed on June 3, 2016, the contents of which are incorporated herein by reference

Field of Invention

[0002] The field of invention relates generally to digital camera technology; and, more specifically to optical flow based auto-focus techniques.

Background

[0003] With the decreased implementation cost of integrating sophisticated processing intelligence into various electronic devices, designers are looking at new image processing paradigms that conceivably could be implemented with an integrated processor and camera. Specifically, new methodologies that possess some computational intensity but at the same time could conceivably be executed by a modestly powerful processor that, e.g., may be integrated with a camera, are particularly of interest.

Summary

[0004] A method is described that includes identifying a set of features, or equivalently reference points, of or on an object in an image, or in an imaged scene. The object may similarly be referred to as a subject of the image or scene. The features or reference points are tracked in the image, that is in a series of frames of the image, which are captured by a camera.

[0005] The method also includes creating a field of vectors for the reference points. The vectors indicate magnitude and direction of change in position of the reference points across more than one frame of the series of image frames. The method further includes identifying existence of out of plane movement of the features or reference points of the object or scene from same radial orientation of the vectors. The method further includes determining an amount of closer/farther movement of the features or reference points of the object or scene to/from the camera from change in distances between a plurality of the reference points. The method further includes adjusting a position of camera's lens in view of the amount of closer/farther movement of the features or reference points of the object or scene to keep the camera focused on the object or scene. [0006] An apparatus is disclosed that includes means for identifying a set of features or reference points of an object or scene, the features being tracked in an image (that is, in a series of image frames) captured by a camera. The apparatus also includes means for creating a field of vectors for the reference points. The vectors indicate magnitude and direction of change in position of the reference points across more than one frame of the image. The apparatus also include means for identifying existence of out of plane movement of the object's features from same radial orientation of the vectors. The apparatus also includes means for determining an amount of closer/farther movement of the object's features to/from the camera from change in distances between a plurality of the reference points. The apparatus also includes means for adjusting a position of a camera's lens in view of the amount of closer/farther movement of the object's features to keep the camera focused on the object. Such an apparatus may be provided by a camera or a suitable part of a camera, or a device comprising a camera such as a smart phone or tablet computer.

[0007] The frames of the series of image frames used for tracking features of the image through time may be consecutive, but of course not every available frame need be used for the tracking process which may therefore use any series of image frames which represent a progression in time of the imaged object or scene, for example every second image frame or an arbitrary selection of subsequent image frames.

Figures

[0008] The following description and accompanying drawings are used to illustrate embodiments of the invention. In the drawings:

[0009] Fii I. 1 shows an object in an image frame;

[0010] Fii ;s. 2a, 2b and 2c depict an object that is moving closer to a camera;

[0011] Fii ;s. 3a and 3b show vector fields from the movement depicted in Figs. 2a through 2c;

[0012] Fii ;s. 4a, 4b and 4c depict an object that is moving farther away from a camera;

[0013] Fii ;s. 5a and 5b show vector fields from the movement depicted in Figs. 4a through 4c;

[0014] Fii I. 6 shows lateral motion of an object;

[0015] Fii I. 7 shows vector fields from the movement of the object in Fig. 6;

[0016] Fii I. 8 shows triangulation of reference points for an object that is moving closer to a camera;

[0017] Fii I. 9 shows triangulation of reference for an object that is moving farther away from a camera;

[0018] Fii I. 10a shows an optical flow based auto focus method;

9 [0019] Fig. 10b shows an apparatus that the method of Fig. 10a can be executed on;

[0020] Fig. 11 shows an embodiment of a computing system.

Detailed Description

[0021] Optical flow is an imaging process that recognizes movement of a recognized object within an image against a background. Because the object can be contrasted against the image's background, the movement of the object can be followed or tracked. Traditionally, however, optical flow processes have been devoted to understanding depth from the detected motion. That is, for instance, the tracking of an object by way of optical flow methods has been used to reconstruct a three dimensional (3D) scene from the object's motion.

[0022] Because optical flow processing methods have largely been applied to the

understanding of depth from the tracked motion of an object, optical flow methods have not heretofore been known to be applied to auto-focusing. Auto-focusing, as is known in the art, involves the movement of a camera lens to keep an object in focus because the distance between the object and the camera has changed.

[0023] Accordingly, the instant application is directed to using optical flow techniques to track the motion of an object that is moving closer to or farther from a camera lens. By tracking such ^out-of-plane^ motion the change in distance between the object and a camera lens can be determined with some precision and, in turn, an adjustment to the camera lens position can be made to keep the object in focus.

[0024] Fig. 1 shows a depiction of a person's face 101 against a background 102. Here, the person's face 101 corresponds to the object that is being tracked. Object recognition processes are generally known in the art and any suitable object recognition process may be used to recognize the person's face 101 against the background 102. The area of the background 102 may generally correspond to the size of a frame of an overall image being captured by a camera. With the object 101 being recognized, or merely with certain features (e.g., eyes, nose, mouth, etc.) being recognized (even if a larger object, such as the person's face, is not recognized), reference points of for the object/features may likewise be readily identified. Here, Fig. 1 shows a plurality of reference points A - O that have been identified both at the perimeter of the object A - H (that circumnavigate the outline of the person's head) and within the object I - O (e.g., eyes, nose, corners of mouth, etc.). Here, again, it is possible that the reference points A - O are identifiable from mere recognition of the specific features of any person's face (eyes, nose, corners of mouth, etc.) without recognition of a particular person's face as a whole (i.e., without recognition of a particular object as a whole). [0025] Figs. 2a through 2c show a sequence of change in the relative positions of the reference points A - O as the person' s face moves closer to the camera. For example, each of Figs. 2a through 2c may correspond to a different image frame captured by a camera at different moments in time. The frames may be immediately consecutive or there may be intervening frames between them.

[0026] Fig. 2a corresponds to an initial position of the face that is farthest away from the camera lens. Fig. 2b shows the face after it has moved closer to the camera lens. Fig. 2c shows the face after it has moved even closer to the camera lens from its position in Fig. 2b. In general, the relative positions of the reference points A - O can be characterized as an "explosion" as they move out radially from a center axis of the face (which approximately runs through reference point L).

[0027] Figs. 3a and 3b show vector fields for the reference points from the sequence of Figs. 2a through 2c. A vector, as is understood in the art, demonstrates both magnitude and direction. The vector field of Fig. 3a shows the magnitude and direction of the change in position of each corresponding reference point as the face moved from the position in Fig. 2a to the position in Fig. 2b. Likewise, the vector field of Fig. 3b shows the magnitude and direction of the change in position of each corresponding reference point as the face moved from the position in Fig. 2b to the position in Fig. 2c.

[0028] Said another way, Fig. 3a shows the divergence of the reference points as the object has moved from its position in Fig. 2a to its position in Fig. 2b. Likewise, Fig. 3b shows the divergence of the reference points as the object has moved from its position in Fig. 2b to its position in Fig. 2c. The divergence when the vectors are pointing radially outward as a consequence of movement of the object toward the lens can be characterized as positive divergence.

[0029] As can be gleaned from the vector diagrams of Figs. 3a and 3b, motion closer to the camera can be identified when reference points of the object explode outward as exemplified by radial movement away from the center of the object. That is, the vectors of the reference points generally point outward radially. Consistent with the explosion analogy, reference points A - H at the periphery of the object (around the perimeter of the person's head) exhibit greater radial motion than reference points I - O within the object.

[0030] That is, the magnitudes of the vectors for the reference points A - H at the perimeter of the person's head are larger than the magnitudes of the vectors for the reference points I - O within the perimeter of the person' s head. In general the magnitudes of the vectors become larger moving out from the center axis of the object. As such, the vectors exhibiting the largest positive divergence essentially identify the periphery of an object that is moving closer to the camera.

[0031] By contrast, Figs. 4a through 4c show a sequence of change in the relative positions of the reference points as the person's face moves farther away from the camera. Fig. 4a corresponds to the initial position of the face as depicted in Fig. 2c. Fig. 4b shows the face after it has moved farther away from the camera lens. Fig. 4c shows the face after it has moved even farther away from the camera lens from its position in Fig. 4b. In general, the relative positions of the reference points can be characterized as an "implosion" as they move radially inward toward the center axis of the person' s face.

[0032] Figs. 5a and 5b show vector fields for the reference points A - O from the sequence of Figs. 4a through 4c. The vector field of Fig. 5a shows the magnitude and direction of the change in position of each corresponding reference point A - O as the face moved from the position in Fig. 4a to the position in Fig. 4b. Likewise, the vector field of Fig. 5b shows the magnitude and direction of the change in position of each corresponding reference point A - O as the face moved from the position in Fig. 4b to the position in Fig. 4c.

[0033] Thus, Fig. 5a shows the divergence of the reference points as the object has moved from its position in Fig. 4a to its position in Fig. 4b. Likewise, Fig. 5b shows the divergence of the reference points as the object has moved from its position in Fig. 4b to its position in Fig. 4c. The divergence when the vectors are pointing inward as a consequence of movement of the object away from the camera lens can be characterized as negative divergence.

[0034] As can be gleaned from the vector diagrams of Figs. 5a and 5b, motion farther away from the camera can be identified when reference points of the object implode inward as exemplified by radial movement into the center of the object. That is, the vectors of the reference points A - O generally point inward radially. Consistent with the implosion analogy, reference points at the periphery of the object A - H (around the perimeter of the person's head) exhibit a greater movement than reference points within the object.

[0035] That is, the magnitudes of the vectors for the reference points A - H at the perimeter of the person's head are larger than the magnitudes of the vectors for the reference points I - O within the perimeter of the person' s head. Again, in general, the magnitudes of the vectors become larger moving out from the center axis of the object and the maximum divergence of the family of vectors (i.e., the vectors having the largest magnitudes) are found at the periphery of the object. As such, the vectors exhibiting the largest negative divergence essentially identify the periphery of an object that is moving farther away from the camera. [0036] As a point of contrast, Figs. 6 and 7 pertain to in-plane motion of the object. Here, Fig. 6 shows the object in three different locations along a horizontal axis as it moves laterally from left to right across the image frame. Fig. 7 shows the corresponding vector vectors for the reference points. Vector field 701 shows the vector field for the object as it moves from the leftmost position to the center position and vector field 702 shows the vector field for the object as it moves from the center position to the rightmost position. Note that all vectors reflect a same amount of positional change in a same direction rather than exploding outward or imploding inward. Additionally, vectors at the leading edge of the object' s motion (e.g., for reference points B, C, D) exhibit positive divergence while vectors at the trailing edge of the object's motion (e.g., for reference points H, G, F) exhibit negative divergence.

[0037] Thus, in plane motion of an object can be differentiated from out of plane motion of the object by calculating vectors for reference points over time that reflect the divergence in the reference points' respective positions over time and recognizing certain characteristics in the vectors as a whole. More specifically, if the family of vectors for an object substantially point in a same direction the object is moving in plane and an auto-focus function need not make any lens position adjustments (the object is not moving closer to or farther from the camera lens).

[0038] By contrast, if the family of vectors for an object substantially point outward radially the object is moving out of plane closer to the camera and an auto-focus function should adjust the camera lens to bring into focus an object that is closer to the camera. Alternatively, if the family of vectors for an object substantially point radially inward, the object is moving out of plane farther away from the camera and an auto-focus function should adjust the camera lens to bring into focus an object that is farther away from the camera.

[0039] After vector tracking of the movement of an object' s reference points reveals that the object is moving out of plane, further processing should determine how far the object has moved out of plane so that a new lens position that keeps the object in focus can be determined. Recall from above that the family of vectors exhibiting maximum divergence generally outline the periphery of the object as it moves out of plane. In an embodiment, triangulation is employed with the reference points within the periphery of the object to determine how far the object has moved out of plane.

[0040] That is, in an embodiment, different reference points are used for different purposes. Specifically, reference points at the periphery of the object A - H are analyzed to flag whether the object is moving out of plane, and, if so, in which direction (closer to the camera or farther from the camera). If the object is deemed to be moving out of plane, reference points within the object I - O are used to determine how far the object has moved. [0041] Thus, in an embodiment, periphery reference points A - H are analyzed to recognize in plane movement (e.g., if some of the periphery reference points exhibit positive divergence while other periphery reference points exhibit negative divergence). If in plane movement is recognized, the camera lens does not need to be repositioned. Likewise, periphery reference points A - H are also analyzed to not only recognize out-of-plane movement but also recognize the direction of the out-of-plane movement (e.g., if all such reference points exhibit positive divergence movement is toward the camera, whereas, if all such reference points exhibit negative divergence movement is away from the camera).

[0042] By contrast, as observed in Figs. 8 and 9, the inner reference points I - O can be used to support a triangulation method of determining how far the object has moved toward or away from the camera. Fig. 8 shows triangulations for determining how far the object has moved toward the camera during the sequence of Figs. 2a through 2c, whereas, Figs. 9 shows triangulations for determining how far the object has moved away from the camera during the sequence of Figs. 4a through 4c.

[0043] Here, Fig. 8 a shows a first triangulation 801 for reference points A - H as the object appears in Figs. 2a, a second triangulation 802 for reference points A - H as the object appears in Figs. 2b and a third triangulation 803 for reference points A - H as the object appears in Figs. 2c.

[0044] Referring to Fig. 8, different ones of reference points A - H are connected by non crossing lines to create a plurality of triangles having respective triangle legs. Specifically, twelve triangle legs are observed in each triangulations 801, 802, 803 of Fig. 8 (IJ, IK, JK, JL, KL, JM, KO, LM, LN, LO, MN, NO). As the object moves closer to the camera, each of the triangle legs grows in length. Thus, the length of the triangle legs can be correlated to the object's distance from the camera, where, larger triangle legs correspond to being closer to the camera and shorter triangle legs correspond to being farther away from the camera.

[0045] Here, some knowledge of the optical design of the camera (e.g., the magnification of the camera lens, the focal distance of the camera lens, etc.) may be needed to precisely correlate a change in triangle leg length to a distance that the object has moved to/from the camera. One of ordinary skill can, for instance, readily develop a mathematical expression that accepts, e.g., average percentage change in triangle leg length across all triangle legs as an input value and provides a distance the object has moved to/from the camera lens as an output value.

[0046] Thus, starting with an object that is known to be in focus, if the object moves closer to the camera the increased length in the triangle legs can be precisely determined and correlated to a precise distance that the object has moved in a direction toward the camera. This distance can then be used to determine a new position for a camera lens to keep the object in focus. [0047] That is, an auto-focus routine can adjust, e.g., a position of a lens within the camera to effectively change the focal length of the camera such that the object remains in focus. Here, the aforementioned mathematical expression that provides object distance movement from triangle leg length change can be further developed to provide appropriate lens movement, from increased triangle leg length change, to reduce focal length distance.

[0048] Similarly, Fig. 9 a shows a first triangulation 901 for reference points A - H as the object appears in Figs. 4a, a second triangulation 902 for reference points A - H as the object appears in Figs. 4b and a third triangulation 903 for reference points A - H as the object appears in Figs. 4c.

[0049] Here, starting with an object that is known to be in focus, if the object moves farther away from the camera the decreased length in the triangle legs can be precisely determined and correlated to a precise distance that the object has moved in a direction away from the camera. This distance can then be used to determine a new position for a camera lens to keep the object in focus.

[0050] If the triangulation calculations determine that the object has moved some distance along the center axis away from the camera, an auto-focus routine will comprehend that the position of the lens needs to be changed a certain amount in a certain direction to keep the object in focus. The auto-focus routine can then adjust, e.g., a the position of a the lens within the camera accordingly. Here, the aforementioned mathematical expression that provides object distance movement from triangle leg length change can be further developed to provide appropriate lens movement, from reduced triangle leg length change, to an appropriate lens to image plane distance.

[0051] Fig. 10a shows an auto-focusing method for a camera as described above. The method includes identifying 1001 a set of features of an object, the features being tracked in an image captured by a camera. The method also includes creating 1002 a field of vectors for the reference points, the vectors indicating magnitude and direction of change in position of the reference points across more than one frame of the image. The method also includes identifying 1003 existence of out of plane movement of the object from same radial orientation of the vectors. The method also includes determining 1004 an amount of closer/farther movement of the object to/from the camera from change in distances between a plurality of the reference points. The method also includes adjusting 1005 a position of a camera's lens in view of the amount of closer/farther movement of the object to keep the camera focused on the object.

[0052] Any of the optical flow based auto-focusing methods described above may be performed, partially or wholly, by a "smart camera" that has processing intelligence integrated with the camera itself (e.g., integrated within a camera package along with lens, photo-detector, etc. or just outside the camera package). For example, a camera may include an integrated processor and memory where the memory stores program code instructions that are executed by the processor to perform any of the optical flow based auto-focusing methodologies described just above. Alternatively or in combination the auto-focusing methodologies may be implemented and/or enhanced with dedicated custom (e.g., logic) circuitry which may be hardwired and/or programmable. Such circuitry again may be integrated with the camera itself (e.g., within the camera package). In still yet other embodiments the optical flow based auto- methodologies, or any portion thereof, may be performed by a main CPU or applications processor that operates out of a main or system memory of a computing system.

[0053] Fig. 10b shows an embodiment of a smart camera that can perform the method of Fig. 10a. As observed in Fig. 10b, the camera 1020 includes a processor 1021 and a memory 1022. The program code for an auto focus routine such as the routine of Fig. 10a is stored in memory

1022 and executed by the processor 1021. The processor 1021 is coupled to an image sensor

1023 which enables the processor to process content being observed by the camera 1020. The processor 1021 is also coupled to a motor 1025 that adjusts the focus. The focus can be adjusted using a motor that changes the position of a focus lens relative to the image sensor 1023, moves a mirror, or mirrors, positioned between a camera lens and the image sensor 1023, or the like. In execution of the method of Fig. 10a, for example, the processor 1021 processes images that are sensed by the image sensor 1023 and, in response thereto, provides input stimuli to the motor 1025 to change the focal length of the lens 1024 and image sensor 1023 in accordance with the auto-focus routine. Other camera embodiments may include more than one lens.

[0054] Fig. 11 provides an exemplary depiction of a computing system. Many of the components of the computing system described below are applicable to a computing system having an integrated camera and associated image processor (e.g., a handheld device such as a smartphone or tablet computer). Those of ordinary skill will be able to easily delineate between the two.

[0055] As observed in Fig. 11, the basic computing system may include a central processing unit 1101 (which may include, e.g., a plurality of general purpose processing cores 1115_1 through 1115_N and a main memory controller 1117 disposed on a multi-core processor or applications processor), system memory 1102, a display 1103 (e.g., touchscreen, flat-panel), a local wired point-to-point link (e.g., USB) interface 1104, various network I/O functions 1105 (such as an Ethernet interface and/or cellular modem subsystem), a wireless local area network (e.g., WiFi) interface 1106, a wireless point-to-point link (e.g., Bluetooth) interface 1107 and a Global Positioning System interface 1108, various sensors 1109_1 through 1109_N, one or more cameras 1110, a battery 1111 , a power management control unit 1112, a speaker and microphone

1113 and an audio coder/decoder 1114.

[0056] An applications processor or multi-core processor 1150 may include one or more general purpose processing cores 1115 within its CPU 1101, one or more graphical processing units 1116, a memory management function 1117 (e.g., a memory controller), an I/O control function (such as the aforementioned peripheral control hub) 1118. The general purpose processing cores 1115 typically execute the operating system and application software of the computing system. The graphics processing units 1116 typically execute graphics intensive functions to, e.g., generate graphics information that is presented on the display 1103. The memory control function 1117 interfaces with the system memory 1102 to write/read data to/from system memory 1102. The power management control unit 1112 generally controls the power consumption of the system 1100.

[0057] Each of the touchscreen display 1103, the communication interfaces 1104 - 1107, the GPS interface 1108, the sensors 1109, the camera 1110, and the speaker/microphone codec 1113,

1114 all can be viewed as various forms of I/O (input and/or output) relative to the overall computing system including, where appropriate, an integrated peripheral device as well (e.g., the one or more cameras 1110). Depending on implementation, various ones of these I/O components may be integrated on the applications processor/multi-core processor 1150 or may be located off the die or outside the package of the applications processor/multi-core processor 1150.

[0058] In an embodiment the camera 1110 uses optical flow based auto focusing techniques as described at length above.

[0059] Embodiments of the invention may include various processes as set forth above. The processes may be embodied in machine-executable instructions. The instructions can be used to cause a general-purpose or special-purpose processor to perform certain processes. Alternatively, these processes may be performed by specific hardware components that contain hardwired logic for performing the processes, or by any combination of programmed computer components and custom hardware components.

[0060] Elements of the present invention may also be provided as a machine-readable medium for storing the machine-executable instructions. The machine-readable medium may include, but is not limited to, floppy diskettes, optical disks, CD-ROMs, and magneto-optical disks, FLASH memory, ROMs, RAMs, EPROMs, EEPROMs, magnetic or optical cards, propagation media or other type of media/machine-readable medium suitable for storing electronic instructions. For example, the present invention may be downloaded as a computer program which may be transferred from a remote computer (e.g., a server) to a requesting computer (e.g., a client) by way of data signals embodied in a carrier wave or other propagation medium via a

communication link (e.g., a modem or network connection).

[0061] In the foregoing specification, the invention has been described with reference to specific exemplary embodiments thereof. It will, however, be evident that various modifications and changes may be made thereto without departing from the broader spirit and scope of the invention as set forth in the appended claims. The specification and drawings are, accordingly, to be regarded in an illustrative rather than a restrictive sense.