SYSTEM AND METHOD FOR DISPLAYING AN IMAGE STREAM

FIELD OF THE INVENTION

The present invention relates to a method and system for displaying and/or reviewing image streams. More specifically, the present invention relates to a method and system for effective displaying of an in vivo image stream.

BACKGROUND OF THE INVENTION

An image stream may be assembled from a series of still images and displayed to a user. The images may be created or collected from various sources, for example, from an in-vivo system with a plurality of imagers.

According to one embodiment, each separated imager may capture images of a lumen such as the gastrointestinal (Gl) tract and a transmitter may transmit the images to an external recording device while the capsule passes through the lumen. Large numbers of images may be collected by each imager for viewing and, for example, may be combined in sequence. An image stream of, for example, 40 minutes in length, containing for example about 4,800 frames, may be presented to the user for review.

SUMMARY OF THE INVENTION

An exemplary embodiment of the present invention provides a system and method for displaying a plurality of image streams, the image streams preferably being produced by an in vivo imaging device such as an ingestible capsule which may include more then one imager, for example, as described in U.S. Patent Serial No. 10/041,541 to Meron et al., titled "SYSTEM AND METHOD FOR WIDE FIELD IMAGING OF BODY LUMENS" assigned to the common assignee of the present application and incorporated herein by reference. A workstation accepts images from each imager and may display the images as image streams. According to some embodiments of the present invention the image streams may simulate, for example a movie or an image stream captured, for example, by an endoscope or a

colonoscope e.g. a movie or an image stream which may display an insertion and retraction of, for example a colonoscope into the colon. According to some embodiments one of the image streams may be displayed in a reverse order while other image streams may be displayed substantially simultaneously on a monitor.

BRIEF DESCRIPTION OF THE DRAWINGS

The principles and operation of the system, apparatus, and method according to the present invention may be better understood with reference to the drawings, and the following description, it being understood that these drawings are given for illustrative purposes only and are not meant to be limiting, wherein:

Fig. 1 shows a schematic diagram of an in-vivo imaging system, according to one embodiment of the present invention;

Fig. 2 shows a portion of a display, according to one embodiment of the present invention; Fig. 3 is a schematic showing how a new image stream may be generated from two image streams, according to one embodiment of the present invention;

Fig. 4 is a schematic illustration of a display of a color bar together with other data captured, according to one embodiment of the present invention; and

Fig. 5 is a schematic illustration of a color bar, according to one embodiment of the present invention.

It should be noted that for simplicity and clarity of illustration, elements shown in the figures have not necessarily been drawn to scale. For example, the dimensions of some of the elements may be exaggerated relative to other elements for clarity. Furthermore, where considered appropriate, reference numerals may be repeated among the figures to indicate corresponding or analogous elements throughout the serial views.

DETAILED DESCRIPTION OF THE INVENTION

In the following description, various aspects of the present invention will be described. For purposes of explanation, specific configurations and details are set forth in order to provide a thorough understanding of the present invention.

However, it will also be apparent to one skilled in the art that the present invention may be practiced without the specific details presented herein. Furthermore, well known features may be omitted or simplified in order not to obscure the present invention. Unless specifically stated otherwise, as apparent from the following discussions, it is appreciated that throughout the specification discussions utilizing terms such as "processing", "computing", "storing", "determining", or the like, refer to the action and/or processes of a computer or computing system, or similar electronic computing device, that manipulate and/or transform data represented as physical, such as electronic, quantities within the computing system's registers and/or memories into other data similarly represented as physical quantities within the computing system's memories, registers or other such information storage, transmission or display devices.

Embodiments of the present invention may include apparatuses for performing the operations herein. Such apparatuses may be specially constructed for the desired purposes, or may comprise general purpose computers selectively activated or reconfigured by a computer program stored in the computers. Such computer programs may be stored in a computer readable storage medium, such as, but is not limited to, any type of disk including floppy disks, optical disks, CD-ROMs, magnetic-optical disks, read-only memories (ROMs), random access memories (RAMs) electrically programmable read-only memories (EPROMs), electrically erasable and programmable read only memories (EEPROMs), magnetic or optical cards, or any other type of media suitable for storing electronic instructions, and capable of being coupled to a computer system bus.

The processes and displays presented herein are not inherently related to any particular computer or other apparatus. Various general purpose systems may be used with programs in accordance with the teachings herein, or it may prove convenient to construct a more specialized apparatus to perform the desired method. It will be appreciated that a variety of programming languages may be used to implement the teachings of the invention as described herein.

Some embodiments of the present invention are directed to a typically swallowable in-vivo device, such as an autonomous swallowable capsule. Other embodiments need not be swallowable or autonomous, and may have other shapes or configurations. Devices according to embodiments of the present invention, including imaging, receiving, processing, storage and/or display units suitable for use with embodiments of the present invention, may be similar to embodiments described in US Patent Application, Pub. No. 2001/0035902 entitled A DEVICE AND SYSTEM FOR IN VIVO IMAGING, and/or in US Patent Number 5,604,531 , each of which are assigned to the common assignee of the present invention and each of which are hereby incorporated by reference. Of course, devices and systems as described herein may have other configurations and other sets of components.

Reference is made to Fig. 1 , which shows a schematic diagram of an in vivo imaging system according to one embodiment of the present invention. The system may include, for example an in-vivo device 40. Device 40 typically may be or may include an autonomous swallowable capsule, but device 40 may have other shapes and need not be swallowable or autonomous. Embodiments of device 40 are typically autonomous, and are typically self-contained. For example, device 40 may be a capsule or other unit where all the components are substantially contained within a container or shell, and where device 40 does not require any wires or cables to, for example, receive power or transmit information. According to one embodiment, device 40 may communicate with an external receiving and display system to provide display of data, control, or other functions. Other embodiments may have other configurations and capabilities. In an exemplary embodiment, the system may comprise a device 40 having one or more imagers, such as imagers 46 and 46' for capturing images, an illumination source 42, for illuminating the body lumen, and a transmitter 41 , for transmitting image and possibly other information to a receiving device. An optical system, including, for example, lenses or mirrors, may aid for example in focusing reflected light onto the imagers 46 and 46'.

According to some embodiments of the present invention transmitter 41 may includes control capability for, for example controlling the various operations of device 40, although control capability or one or more aspects of control may be included in a separate component. Transmitter 41 is typically an ASIC (application specific integrated circuit), but may be of other constructions; for example, transmitter 41 may be a processor executing instructions. Device 40 may include a processing unit separate from transmitter 41 that may, for example, contain or process instructions.

In this embodiment the imagers, such as imagers 46 and 46' may be positioned such that they capture images of different portions of the body lumen. A combined image of the images captured by the different imagers may show all the individually imaged portions, thereby covering a wide field.

Preferably, located outside the patient's body in one or more locations, are an image receiver 12, preferably including an antenna or antenna array, an image receiver storage unit 16, a data processor 14, a data processor storage unit 19, and an image monitor 18, for displaying, inter alia, the images recorded by device 40. According to one embodiment, the image receiver 12 and image receiver storage unit 16 are small and portable, and are worn on the patient's body during recording of the images. According to embodiments of the present invention, data processor storage unit 19 includes an image database 21.

According to one embodiment of the present invention, data processor 14, data processor storage unit 19 and monitor 18 are part of a personal computer or workstation which includes standard components such as processor 14, a memory, a disk drive, and input-output devices, although alternate configurations are possible, and the system and method of the present invention may be implemented on various suitable computing systems.

Data processor 14 may include any standard data processor, such as a microprocessor, multiprocessor, accelerator board, or any other serial or parallel high performance data processor. Image monitor 18 may be a computer screen, a conventional video display, or any other device capable of providing image or other data

Preferably, an imager, for example imager 46 is a suitable CMOS camera, such as a "camera on a chip" type CMOS imager specified by Given Imaging Ltd. of Israel and designed by Photobit Corporation of California, USA. In alternate embodiments, the imager 46 may be another device, for example, a CCD. The illumination source 42 may be, for example, one or more light emitting diodes, or another suitable light source.

Fig. 2 depicts a portion of a display according to an embodiment of the present invention. Referring to Fig. 2, the display 200 is in multiple image streams mode. The display 200- may be displayed on, for example, image monitor 18. Typically, the display 200 includes a set of image windows 202 and 202 ^! , for displaying image streams, and a set of controls 204. The controls 204 may include, for example, a toggle control 206, allowing the user, for example to toggle between a multiple image streams mode and a single image stream mode. In the multiple image streams mode the user may control all the image streams displayed in image windows 202 and 202' simultaneously by controls 204. In the single stream mode, controls 204 may be used to control one of the image streams, and the toggle control 206 may allow a user to select the image stream to be controlled by the controls 204. The controls 204 may also include, for example, conventional video controls, such as pause, stop, play, fast-forward, reverse, etc. In a typical embodiment, if the system is in a multiple image streams mode, the controls 204 may act on all image streams simultaneously; in alternate embodiments, other methods may be used, for example the controls 204 may act on a single image stream.

According to some embodiments of the present invention, the different image streams may be placed in different configurations on a viewing screen. For example, rather than horizontally, the image streams may be arranged vertically or diagonally.

In certain embodiments of the present invention, more than one image stream may be collected. For example, as shown in Fig. 1 an in-vivo imaging device may include more than one imager (or one imager) collecting multiple image streams — possibly by including an imager or lens system in more than one location on the in-vivo imaging device. The imagers, such as imagers 46 and 46' may be

arranged, for example, at either end of device 40, or at the same end of the device, in slightly different positions or different angles. Each imager, for example imager 46, may capture images and transmit the images via the transmitter 41 or via separate transmitters. Typically, each imager has an optical system associated with it.

According to some embodiments of the present invention, one or two image streams captured for example by the imagers 46 and 46' may simulate, for example a movie or an image stream captured, for example by an endoscope or a colonoscope e.g. a movie or an image stream which may display a simulation of an insertion and/or retraction of, for example an colonoscope into a colon.

In such a case, an embodiment of the system and method of the present invention may display a first image stream in a reversed order, for example all the images captured by a first imager, for example imager 46 which may be located at the front of the in vivo device may be displayed in a reversed order. According to some embodiments of the present invention, a second image stream e.g. which may represent all images captured from another imager, for example imager 46' located in the rear of the in vivo device, may be displayed after the first image stream. According to some embodiments a device having only one imager and/or one viewing angle may be used to obtain images.

Reference is now made to Fig. 3, a schematic showing how a new image stream 320 may be generated from two image streams, such as original image streams A and B, and may be displayed according to a predetermined criteria, and/or criteria of interest, according to some embodiments of the present invention. According to one embodiment of the present invention, an ID (Identification) number may be assigned to each image, for example by a recorder during a recording process. Typically, a recorder receives the first image from image stream A and assigns to that image the ID number 1 , and then records it. To simplify and further clarify this process the first image will hereafter be referred to as image 1a, representing the first recorded image from an image stream A. In the next step the recorder receives the first image from image stream B and assigns to it the ID

number 2b (e.g. second image received by the recorder and the first image from image stream B).

According to some embodiments of the present invention, image stream A (which may include images 1a, 3a, 5a, 7a, 9a ...Xa) may represent for example all the images captured by a single imager, for example imager 46 which may be located at the front of the in vivo device, and image stream B (which may include images 2b, 4b, 6b, 8b, 10b etc.) represents all the images captured from another imager, for example imager 46' located in the rear of the in vivo device. The first images of image stream A and image stream B may represent images captured, for example in the cecum, the next images of image streams A and B may represent image captured in the colon and the last images may represent images captured in the rectum.

According to some embodiments of the present invention the image streams A and B may be reorganized into a new image stream e.g. image stream 320. The new order of the images in the image stream 320 may simulate, for example a movie or an image stream captured for example by an endoscope or a colonoscope. For example the first images of image stream 320 (e.g. image Xa, which is the last image in image stream 310A) may simulate the insertion of an endoscope into the body (e.g. images of the rectum) the next images of image stream 320 (e.g. images 5a, 3a, 1a and 2b) may simulate the insertion of the endoscope into the colon. The last images of image stream 320 (e.g. images 2b- Xb) may display a simulation of the retraction of an endoscope from the colon (e.g. images of the cecum, the colon and the rectum). According to other embodiments images from image stream 310 B shown in their reverse order may simulate entrance into the body through the rectum and images from image stream 310A shown in their original order may simulate the retraction of a device through the colon to the rectum.

According to some embodiments a combined image stream 320 may include images from a single original image stream, wherein the images from the original images stream may be arranged in an appropriate arrangement to create the illusion of a certain path taken by the imaging device.

Reference is now made to Fig. 4 showing a display and/or a graphical user interface 400 for displaying data captured in-vivo. Display 400 may include a summarized graphical presentation 420 of an in-vivo data stream, for example, a color bar. Typically, the graphical presentation 420 may be a fixed presentation displayed alongside a streaming display of a data stream 410, for example, the image stream 320 of Fig. 3. In other embodiments of the present invention, graphical presentation 420 may be displayed separately. The graphical presentation 420 may include a series of colors, a series of colored areas, or a series of patterns, image items, images or pixel groups (e.g., a series of stripes 422 or areas of color arranged to form a larger bar or rectangular area), where each color in the series 422 may be associated with and/or correspond to an element or a group of elements in the original data stream. For example, each colored stripe 422 may correspond to an image or a group of images from a data stream 410 e.g. image stream 320. Image units other than stripes (e.g., pixels, blocks, etc.) may be used, and the image units may vary in a dimension other than color (e.g., pattern, size, width, brightness, animation, etc). One image unit (e.g., a stripe 422) may represent one or more units (e.g., image frames) in the original data stream. Typically, the series of, for example, colors in the bar may be arranged in the same sequence or order in which the data stream, for example, the images or groups of images may typically be displayed. In one embodiment of the present invention, pointing at a stripe in a graphical presentation 420 may advance the image stream to the frames corresponding to that stripe.

In one example, a marker or indicator 450 may progress across or along the graphical presentation 420 as the substantially corresponding datum in data stream 410 (e.g., video) may be concurrently displayed to indicate the correspondence between the graphical presentation 420 and the data stream 410. In other embodiments of the invention, the presentation may be of a shape other than a bar, for example a circle, oval, square, etc. According to other embodiments, the presentation may be in the form of an audio tract, graph, and other suitable graphic presentations.

An indicator 450 such as a cursor may advance along the time bar 430 and graphical presentation 420 as the image stream 410 is scrolled on the display 400. In one example, control buttons 440 may be included in the display that may allow the user to, for example, fast-forward, rewind, stop play or reach the beginning or end of, for example, an image stream 410. In other embodiments of the present invention, a user may control the display of a data stream 410, for example, by altering the start position of the streaming display, e.g. skipping to areas of interest, by moving the position of indicator 450, for example with a mouse or other pointing device. In other embodiments of the present invention, a user and/or health professional may insert indications or markers such as thumbnails to mark location along the image stream for easy access to those locations in the future.

Fig. 5 is a schematic illustration of a tissue color bar according to an embodiment of the present invention. Graphic tissue presentation 520 may have been obtained through image processing of a stream of images obtained, for example, from imagers 46 and 46 ¹ imaging the tissue of the Gl tract. Other lumens may be sensed, and other modalities (e.g., temperature) may be sensed. The graphic tissue presentation 520 may be horseshoe shaped and may represent, for example, a compressed and perhaps smoothed version of an image stream, such as the image stream 320 of Fig. 3. For example the left section of the graphic tissue presentation 520 e.g. section 530 may represent for example a simulation of an insertion of a colonoscope into a body lumen. According to some embodiments of the present invention section 530 may represent for example images which have been captured by imager 46 of Fig. 1. The first group of images in section 530 e.g. group of images 531 , may represent images captured in the rectum. The next group of images in section 530 e.g. group of images 532 may represent images captured in the colon, and the last group of images in section 530 e.g. group of images 533, may represent images captured in the cecum. According to some embodiments of the present invention the right section of the tissue graphical presentation 520 e.g. section 530' may represent for example a simulation of a retraction of a colonoscope from a body lumen. Section 530 may represent for example images which have been

captured by imager 46' of Fig. 1. The first group of images in section 530 e.g. group of images 531', may represent images captured in the rectum. The next group of images in section 530' e.g. group of images 532', may represent images captured in the colon, and the last group of images in section 530' e.g. group of images 533', may represent images captured in the cecum.

According to some embodiments a time line that corresponds with the image stream may be presented on the monitor, for example, as a time bar. According to some embodiments a user may start viewing an image stream at the end . the image stream will then progress from the anus to the secum and the time bar may be made to follow the image stream, for example like this:

[anus] 1 :00 2:00 3:00. When the user reaches and possibly marks the cecum, the time bar may mirror itself and may show a mirror image of itself to the right of the marking, like this: [anus] 1 :00 2:00 3:00 [cecum] 3:00 2:00 1 :00 [anus]. From the cecal entrance, the video player may automatically jump between imagers or image streams as it runs during the showing of the moving image stream. This may give the user the impression that he is watching one video or movie which starts and ends at the anus, rather than 2 essentially repeated movies.

Embodiments of the invention provide a method for viewing a movie of images captured by an imaging capsule. According to some embodiments the method include the steps of presenting to a viewer on a screen a single movie which includes a reversed image stream in combination with the image stream shown in its original direction. According to some embodiments the single movie may include two or more image streams captured from two or more angles (e.g., two image streams captured from two opposing sides of a swallowable imaging capsule). Typically one of the image streams may be presented in a reversed order together with another of the image streams that is presented in its original order.

The method may further include the step of displaying an indication line or bar which corresponds to the image stream being shown. Examples of indication lines may include a time line, a color line, a location display and other suitable displays. According to some embodiments a movie may be made to be played in

reverse and the indication line may mirror itself once a certain, predetermined image is presented (e.g., images of the cecum).

The foregoing description of the embodiments of the invention has been presented for the purposes of illustration and description. It is not intended to be exhaustive or to limit the invention to the precise form disclosed. It should be appreciated by persons skilled in the art that many modifications, variations, substitutions, changes, and equivalents are possible in light of the above teaching. It is, therefore, to be understood that the appended claims are intended to cover all such modifications and changes as fall within the true spirit of the invention.