Inventors: Kang-Huai Wang, Chenyu Wu and Yi Xu CROSS REFERENCE TO RELATED APPLICATIONS

[0001] The present invention is a continuation-in-part and claims priority to PCT Patent Application, Serial No. PCT/US 16/52975, filed on September 22, 2016, which claims priority to U.S. Patent Application, Serial No. 14/884,788, filed on October 16, 2015. The PCT Patent Application and the U.S. Patent Application are hereby incorporated by reference in their entireties.

FIELD OF THE INVENTION

[0002] The present invention relates to the endoscope for capturing images of human gastrointestinal (GI) tract for diagnosis purpose. In particular, the endoscope is enabled to measure distance of objects in the field of view of the camera. The distance information can be used subsequently to process the image sequence captured, such as measuring a size of an object of interest or stitching the image sequence to reduce viewing time.

BACKGROUND AND RELATED ART

[0003] Devices for imaging body cavities or passages in vivo are known in the art and include endoscopes and autonomous encapsulated cameras. Endoscopes are flexible or rigid tubes that pass into the body through an orifice or surgical opening, typically into the esophagus via the mouth or into the colon via the rectum. An image is formed at the distal end using a lens and transmitted to the proximal end, outside the body, either by a lens-relay system or by a coherent fiber-optic bundle. A conceptually similar instrument might record an image electronically at the distal end, for example using a CCD or CMOS array, and transfer the image data as an electrical signal to the proximal end through a cable. Endoscopes allow a physician control over the field of view and are well-accepted diagnostic tools.

[0004] Capsule endoscope is an alternative in vivo endoscope developed in recent years. For capsule endoscope, a camera is housed in a swallowable capsule, along with a radio transmitter for transmitting data, primarily comprising images recorded by the digital camera, to a base-station receiver or transceiver and data recorder outside the body. The capsule may also include a radio receiver for receiving instructions or other data from a base-station transmitter. Instead of radio-frequency transmission, lower-frequency electromagnetic signals may be used. Power may be supplied inductively from an external inductor to an internal inductor within the capsule or from a battery within the capsule.

[0005] An autonomous capsule camera system with on-board data storage was disclosed in the US Patent No. 7,983,458, entitled "In Vivo Autonomous Camera with On-Board Data Storage or Digital Wireless Transmission in Regulatory Approved Band," granted on July 19, 2011. The capsule camera with on-board storage archives the captured images in on-board non-volatile memory. The capsule camera is retrieved upon its exiting from the human body. The images stored in the non-volatile memory of the retrieved capsule camera are then accessed through an output port on in the capsule camera.

[0006] When the endoscope is used for imaging the human GI tract, one of the primary purposes is to identify any possible anomaly. If any anomaly is found, it is further of interest to determine characteristics of the anomaly, such as the size of the anomaly. The captured images will be examined by medical profession for examination or diagnosis. The number of images captured is typically 25,000 or more. It will require a long reviewing time to look through the images even by skilled professionals. Accordingly, image stitching has been used to reduce the number of images to be viewed. For example, in PCT Patent Application Publication, Serial No. WO2014/193670 A2, published on Dec. 4, 2014, image stitching for images captured using a capsule camera is disclosed. It is desirable to develop methods or apparatus that are capable to further improve the efficiency of image stitching. BRIEF SUMMARY OF THE INVENTION

[0007] A method and apparatus for capturing images of a scene using a capsule camera are disclosed. After the capsule camera is swallowed by a patient, an image sequence is captured using the capsule camera when the capsule camera travels through a human gastrointestinal tract. Also, distance information associated with objects of the image sequence with respect to the capsule camera is captured when the capsule camera travels through the human gastrointestinal tract. Both the image sequence and the distance information are outputted.

[0008] The association information between the distance information and related images of the image sequence is outputted. The association information may correspond to frame numbers or capture times of the related images of the image sequence. The distance information can be determined based on Time-of-Flight or phase shift using a light or ultrasound source. The distance information may also be derived from structured-light images by projecting structured light to one or more objects in a field of view of the capsule camera and capturing the structured-light images using the capsule camera.

[0009] The present invention also discloses a method of determining the size of an object of interest in an image. The image sequence captured by a capsule camera along with the distance information is received. The size of an object of interest in a selected image in the image sequence can be determined based on the selected image and the distance information. The size of the object of interest is determined according to the image size of the object of interest in a selected image scaled by a ratio of object distance to the capsule camera and focal length of the capsule camera. The image size of the object of interest in the selected image can be measured in terms of a number of pixels of object of interest in the selected image. The distance information can be in a form of structured-light images that are captured by projecting a structured light onto one or more objects in a field of view of the capsule camera and capturing the structured-light images using the capsule camera, and the distance information is further derived from the structured-light images before the distance information is used for said determining the size information of the object of interest. [0010] The present invention further discloses a method of stitching the image sequence utilizing information including the distance information to generate a stitched image sequence. Again, the distance information can be in a form of structured-light images. In one

embodiment, the distance information is used to scale the objects of the image sequence for stitching the image sequence. In another embodiment, the distance information is used to adjust image intensities of the image sequence for stitching the image sequence.

[0011] BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Fig. 1 illustrates an example of relationship among the size of an object, the size of the corresponding object image, the object distance and the focal length of the camera.

[0013] Fig. 2 A and Fig. 2B illustrate an example of the different sizes of object images for a same object in two images captured at two different object distances.

[0014] Fig. 3 illustrates an exemplary flowchart for capturing an image sequence along with distance information according to an embodiment of the present invention.

[0015] Fig. 4 illustrates an exemplary flowchart for determining a size of an object of interest in the image sequence based on the image sequence and the distance information according to an embodiment of the present invention.

[0016] Fig. 5 illustrates an exemplary flowchart for stitching an image sequence utilizing information including the distance information to generate a stitched image sequence according to an embodiment of the present invention.

DETAILED DESCRIPTION OF THE INVENTION

[0017] It will be readily understood that the components of the present invention, as generally described and illustrated in the figures herein, may be arranged and designed in a wide variety of different configurations. Thus, the following more detailed description of the embodiments of the systems and methods of the present invention, as represented in the figures, is not intended to limit the scope of the invention, as claimed, but is merely representative of selected embodiments of the invention. References throughout this specification to "one embodiment," "an embodiment," or similar language mean that a particular feature, structure, or characteristic described in connection with the embodiment may be included in at least one embodiment of the present invention. Thus, appearances of the phrases "in one embodiment" or "in an embodiment" in various places throughout this specification are not necessarily all referring to the same embodiment.

[0018] Furthermore, the described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments. One skilled in the relevant art will recognize, however, that the invention can be practiced without one or more of the specific details, or with other methods, components, etc. In other instances, well-known structures, or operations are not shown or described in detail to avoid obscuring aspects of the invention. The illustrated embodiments of the invention will be best understood by reference to the drawings, wherein like parts are designated by like numerals throughout. The following description is intended only by way of example, and simply illustrates certain selected embodiments of apparatus and methods that are consistent with the invention as claimed herein.

[0019] Endoscopes are normally inserted into the human body through a natural opening such as the mouth or anus. Therefore, endoscopes are preferred to be small sizes so as to be minimally invasive. As mentioned before, endoscopes can be used for diagnosis of human gastrointestinal (GI) tract. The captured image sequence can be viewed to identify any possible anomaly. If any anomaly is found, it is of interest to identify the characteristics of the anomaly, such as the size. Accordingly, an invention of the present invention discloses an endoscope including a distance measuring means to measure the object distances between the camera and various locations of an object in the field of view of the camera.

[0020] There are various known means for measuring the distance between the camera and various locations of an object in the field of view of the camera. For example, there is a class of distance measuring devices that determine the distance based on ToF (Time of Flight) or phase shift of a light source. The light source may be a laser or LED (Light Emitting Diode). A light sensor is used to detect the returned light. The time difference or phase difference between the emitted light from the light source and the received light by the light detector is used to determine the distance. Ultrasonic wave is also a signal source that can be used to measure the distance between an object and the camera for the intended GI imaging application. The distance measuring means is well-known in the field and various literatures describing the distance measure based on ToF or phase shift using light or ultrasound sources are readily available. Therefore, details for distance measuring means based on ToF or phase shift using light or ultrasound sources are omitted in this disclosure.

[0021] If a light source is used to measure the distance, the light for measuring the distance may interfere with the flash light illuminating the GI tract during image capture. In this case, the light for distance measuring and the flash light for image capture will not be applied at the same time, or at least one of the light sources needs to be substantially dimmed. The distance information can be stored separately or stored with an associated image. The distance information can be captured before or after an associated image. If the distance information is stored separately, the related information (named as association information in this disclosure) for the associated image will also be stored so that the distance information can be properly used. The related information can be capture time, frame time or frame number of the associated image. If the ultrasound is used to measure the distance, the distance measuring using ultrasound and image capturing by applying the flash light to illuminating the GI tract may occur at the same time.

[0022] While the distance measuring means based on ToF or phase shift using light or ultrasound is well known, to fit such distance measuring means into an endoscope is a challenging task and costly with larger size not suitable for endoscope application.

Accordingly, other distance measuring means are based on image processing of images captured using an image sensor. [0023] For example, one technique for capturing depth information is to use a color filter placed on top of selected sensor pixels with the passband reasonably narrow and capture the color information and depth information simultaneously. The environment light sources with spectrum in the filter passband will cause negligible amount of energy projected onto the sensor. For the case of RGB pixels, a fourth type of pixels may be added to capture light with the spectrum in the passband of the filter placed on top of these pixels. Then, the structured light that has the spectrum substantially in the passband can be projected onto the scene. However this approach will reduce the spatial resolution of the images or video captured using such image sensor and require the use of an unconventional color filter.

[0024] Another technique is to obtain the depth information as well as 3D topology by projecting structured-light patterns that are visible in the RGB sensors. However the real time image and/or video will be confounded by the structured light superimposed on it. When a structured image is captured, the depth or shape information for objects in the scene is derived. The depth or shape information is then assumed by the image or images captured shortly before or after the structured light image. Since the regular images are captured by a capsule endoscope at a relatively slow frame rate (e.g. 5 frames per second), the scene corresponds to the image captured using the structured light and the scene corresponds to a regular image may be noticeably different due to the endoscope movement or the peristaltic motion of the intestines. In order to improve the accuracy of the derived depth information from the structured light images, the structured images with shortened frame period is disclosed in U.S. Patent Application, Serial No. 14/884,788, filed on October 16, 2015. Since the structured light image is closer in time with the corresponding regular image, the depth information derived should be more accurate than that derived based on a structured-light image with a longer frame period.

[0025] When the distance measuring means via structured light is used, the depth (i.e., distance) information will be derived from the structured-light images. In other words, the raw distance information is stored in the form of structured-light images. In this case, the distance information (i.e., the structured-light images) can be stored separately or stored with an associated image taken under regular light. The distance information can be captured before or after an associated image. If the distance information is stored separately, the related information (i.e., association information) for the associated image will also be stored so that the distance information can be properly used. Techniques to derive the depth information from the structured-light images are known in the field. Details of the depth derivation from the structured-light images are omitted in this disclosure.

[0026] In an endoscope, the focal length is known by design. If the distance (also named as object distance in this disclosure) between an object and the camera can be determined, the dimensions of an object can be determined simply using geometry. Fig. 1 illustrates a simplified example of object dimension determination based on object-camera distance. In a camera system, the image sensor is placed at the focal plane 120 behind the lens 110. The camera can capture a scene within the field of view extending an angle a. The focal length /is the distance between the lens and the image sensor. The focal length often is fixed for endoscopic applications and is known by design. However, when a capsule endoscope travels through the GI tract, the object distance D varies depending on the location of the capsule endoscope and its relative angles with respect to the GI wall being imaged. If the distance D is known, the dimension of an object can be determined from the captured image by measuring the size of the object image in the image. For example, if an object 130 with height H is at distance D from the camera, the object image height H can be derived from the object image height h in the image according to:

[0027] In the above equation, h is measured from the image, the focal length /is known by design, and the distance / ⁾ is determined by a selected distance measuring means as mentioned above. Accordingly, if the distance can be determined, the object dimensions can be derived. The object size in the image can be measured in physical dimension. However, the image is captured digitally and the size measurement may be more convenient in terms of the number of pixels. Since the physical dimension of image sensor surface and the optical footprint are known. Also, the number of pixels is known (e.g. 320x240). Therefore, the object image size in the image can be measured in a number of pixels and converted physical object image size in the image.

[0028] As shown above, the object image size in the image depends on the actual object size and its distance from the camera. A smaller object at a closer distance may appear to have the same size in the image as a larger object at a farther distance. For example, the object 140, which is smaller but closer than object 130, appears to have the same height as object 130 in the image. Therefore, the distance is crucial information for determining the object size.

Accordingly, the distance measuring means disclosed above enables object size determination based on the images captured using an endoscope.

[0029] Distance information is also useful for image stitching. In an ideal situation with a solid object model, object size variations in the captured images may be implicitly taken care by the registration process. A corresponding object in different images can be identified and registered. The different object sizes in different images due to distance variations are presumably taken into consideration by the registration process. The object having a difference size in a target frame will be matched with the corresponding object in the reference frame. Likely, a global motion model will be applied to the target image to scale the object so that the image can be scaled and stitched properly. Nevertheless, the images associated with objects in the GI tract environment are usually far from the ideal solid object models. Furthermore, the iteration process may not always converge or converge to local minima when variables such as distance are involved in the optimization process. As is known in the field, the iteration process is usually used as part of the whole registration process. In one embodiment, the distance information is used for scaling. In particular, the distance information is used to assist the image registration to improve the registration accuracy.

[0030] Fig. 2 A and Fig. 2B illustrate an example of the different sizes of object images for a same object in two images captured at two different object distances. In Fig. 2A, illustration 210 corresponds to the case that the capsule 211 is at a farther distance from an object of interest 213 of the GI tract 212. Image 220 corresponds to the image captured for illustration 210. In Fig. 2B, illustration 230 corresponds to the case that the capsule 211 is at a closer distance from an object of interest 213 of the GI tract 212. Image 240 corresponds to the image captured for illustration 230. Since image 240 is capture with the camera closer to the GI wall. Therefore, the object image in image 240 appears larger than the object image size in image 220. Therefore, the distance information can be used to scale the object in these two images.

[0031] In the GI tract environment, images are always captured using a light source to illuminate the field of view. When the camera is closer to an intestine wall, the object being imaged will be brighter and the image intensities will be higher. On the other hand, when the camera is farther from an intestine wall, the object being imaged will be dimmer and the image intensities will be lower. Therefore, the overall intensities of an image will depend on the distance between the object in the field of view and the camera. Since the object-camera distance is typically very short for the GI tract environment, the variation in the overall intensities of an image will be rather large. Such large intensity variation may degrade the registration performance and consequently lower the stitching performance.

[0032] Accordingly, in another embodiment of the present invention, the image intensities for two images to be registered will be adjusted according to the distance. The pixel intensities are roughly proportional to the distance square inversely or another functional form. The relation between the pixel intensities and the distance can also be tabulated instead of being represented in a functional form. The intensities of an image at a closer distance can be adjusted down to match with those of another image at a farther distance. Alternatively, the intensities of an image at a farther distance can be adjusted up to match with those of another image at a closer distance. After intensity adjustment to compensate the variation due to different distances, the registration and image stitching should perform better.

[0033] Fig. 3 illustrates an exemplary flowchart for capturing an image sequence along with distance information according to an embodiment of the present invention. The capsule device is administered to a patient in step 310. An image sequence is captured using the camera in the capsule device when the capsule device travels through a human gastrointestinal tract in step 320. The structured-light images are captured using the camera by projecting structured light to one or more objects in a field of view of the camera when the capsule device travels through the human gastrointestinal tract in step 330, where the structured-light images are interleaved with regular images in the image sequence. The distance information is derived from the structured-light images in step 340, where the distance information is associated with objects of the image sequence with respect to the camera. The image sequence is outputted in step 350. The distance information is outputted in step 360. The distance information extracted from structure light images implies that the respective distance information is determined at more than one point in the image or field of view. It's the intention of the present invention to include one or more points in image.

[0034] Fig. 4 illustrates an exemplary flowchart for determining a size of an object of interest in the image sequence based on the image sequence and the distance information according to an embodiment of the present invention. An image sequence captured by the capsule camera when the capsule camera travelled through a human gastrointestinal tract is received in step 410. The distance information with respect to the capsule camera associated with objects of a selected image in the image sequence captured by the capsule camera when the capsule camera travelled through the human gastrointestinal tract is received in step 420. The size of an object of interest in the selected image is determined based on contents of the selected image and the distance information in step 430. The size information regarding the size of the object of interest is outputted in step 440.

[0035] Fig. 5 illustrates an exemplary flowchart for stitching an image sequence utilizing information including the distance information to generate a stitched image sequence according to an embodiment of the present invention. An image sequence captured by the capsule camera when the capsule camera travelled through a human gastrointestinal tract is received in step 510. The distance information with respect to the capsule camera associated with objects of the image sequence captured by the capsule camera when the capsule camera travelled through the human gastrointestinal tract is also received in step 520. The image sequence is stitched utilizing information including the distance information to generate a stitched image sequence in step 530. The stitched image sequence is outputted in step 540.

[0036] The above description is presented to enable a person of ordinary skill in the art to practice the present invention as provided in the context of a particular application and its requirements. Various modifications to the described embodiments will be apparent to those with skill in the art, and the general principles defined herein may be applied to other embodiments. Therefore, the present invention is not intended to be limited to the particular embodiments shown and described, but is to be accorded the widest scope consistent with the principles and novel features herein disclosed. In the above detailed description, various specific details are illustrated in order to provide a thorough understanding of the present invention. Nevertheless, it will be understood by those skilled in the art that the present invention may be practiced.

[0037] The invention may be embodied in other specific forms without departing from its spirit or essential characteristics. The described examples are to be considered in all respects only as illustrative and not restrictive. The scope of the invention is, therefore, indicated by the appended claims rather than by the foregoing description. All changes which come within the meaning and range of equivalency of the claims are to be embraced within their scope.