Selection of images for optical examination of the cervix

FIELD OF THE INVENTION

The invention relates to the field optical examination of the cervix, particularly to the field of colposcopy.

BACKGROUND OF THE INVENTION

Cancer arising from the cervix is the number one cancer in women in many countries. About 30% of cancers in women are due to cervical cancer with more than

100,000 new cases diagnosed every year, e.g., in India.

After a positive screening test for cervical cancer, colposcopic examination is routinely used as the second diagnostic step by gynecologists for identification of abnormal areas of the cervix. A colposcope is a low-power, stereoscopic, binocular field microscope with a powerful light source used for magnified visual examination of the uterine cervix to help in the diagnosis of cervical cancer.

A routine test for cervical cancer applied worldwide, and in which a colposcope is used, involves the reaction of tissue to the administration of acetic acid and iodine solution to the cervix.

A colposcope is used to identify visible clues suggestive of abnormal tissue. It functions as a lighted binocular microscope to magnify the view of the cervix, vagina, and vulvar surface. Low power (2x to 6 ^X ) may be used to obtain a general impression of the surface architecture. Medium (8 ^X to 15 ^x ) and high (15 ^x to 25 ^x ) powers are utilized to evaluate the vagina and cervix. The higher powers are often necessary to identify certain vascular patterns that may indicate the presence of more advanced precancerous or cancerous lesions. Various light filters are available to highlight different aspects of the surface of the cervix.

Acetic acid (usually 3 - 5 %) is applied to the cervix by means of, e.g., cotton swabs, or spray. Areas with a high risk of neoplasia, or cancer, will appear as varying degrees of whiteness, because acetowhiteness correlates with higher nuclear density. The term "acetowhiteness" is used in contradistinction to areas of hyperkeratosis or leukoplakia which appear white before the application of acetic acid. The transformation zone is a critical area on the cervix where many precancerous and cancerous lesions most often arise. The ability to see the transformation zone and the entire extent of any visualized lesion determines whether an adequate colposcopic examination is attainable.

Areas of the cervix which turn white after the application of acetic acid or have an abnormal vascular pattern are often considered for biopsy. If no lesions are visible, an iodine solution may be applied to the cervix to help highlight areas of abnormality.

After a complete examination, the colposcopist determines the areas with the highest degree of visible abnormality and may obtain biopsies from these areas using a long biopsy instrument. Most doctors and patients consider anesthesia unnecessary, however, some colposcopists now recommend and use a topical anesthetic such as lidocaine or a cervical block to diminish patient discomfort, particularly if many biopsy samples are taken.

Extensive training is needed to correctly interpret a colposcope test according to the above protocol. In emerging markets like India and China lack of trained resources and expertise limit the usage of this effective diagnostic tool. The same situation applies for industrialized countries, where qualified medical personnel are in short supply.

SUMMARY OF THE INVENTION

The invention provides for a medical imaging system, a computer program product and a method of operating a medical imaging system in the independent claims.

Embodiments are given in the dependent claims.

Embodiments of the invention may provide a means of addressing the lack of qualified persons for interpreting a colposcope test by providing an automated system for selecting a diagnostic image.

In some embodiments video data from various phases of the examination is classified and satisfactory images are selected using feature correspondence. In other embodiments of the invention the quality of the diagnostic image is evaluated to see if it is satisfactory for performing a diagnosis or not. Both embodiments may enable less well trained health care practitioners to obtain diagnostic images which are suitable for diagnosis and then may be subsequently passed on to better trained health care practitioners who are able to provide a diagnosis. Current colposcopy examinations are subjective and depend on the knowledge and experience of a gynaecologist for the interpretation. It is thus an object of the present invention to eliminate subjectivity from the process. It is another object of the present invention to increase usability of colposcopy by reducing the learning curve and assisting in diagnosis. It is another object of the present invention to provide better confidence to the user (gynaecologist). It is still another object of the present invention to provide a quantitative measure for the degree of cancer severity.

These objects are achieved by a system or a method according to the present invention.

According to the invention, a system for optical examination of the cervix is provided, said system comprising optical magnification means, illumination means, dispensing means for administration of at least one stimulation and/or contrasting agent, imaging means and image processing means. Said image processing means further comprises key frame extraction means, optionally, glare removal means, object detection means (also called "object detector") and opacity change detection means.

In a preferred embodiment, the system further comprises operator interfacing means for data input and data output. Such interfacing means is, for example, a display screen, a keyboard, a mouse, a touch screen, a touchpad, a joystick or likewise.

Preferably, said stimulation and/or contrasting agents are selected from the group consisting of acetic acid and/or iodine solution, preferably Lugol's solution or

Schiller's solution. As becomes evident from the specification of the present invention, the skilled person may find alternative stimulating and/or contrasting agents in the respective literature by routine work. Such alternative stimulating and/or contrasting agents are thus comprised by the scope of protection provided by the present invention.

In another preferred embodiment, the imaging means comprises either a digital imaging device, or a non-digital camera and a frame grabber. Said digital imaging device is preferably a digital camera. Said digital or non-digital camera comprises, preferably, a CCD (Charge Coupled Device) or a CMOS (Metal Oxide Semiconductor) camera.

In another preferred embodiment, the optical magnification means is a colposcope. A colposcope is a low-power, stereoscopic, binocular field microscope with a powerful light source, which is used for magnified visual examination of the uterine cervix to help in the diagnosis of cervical cancer. In another preferred embodiment, the system further comprises a computer workstation for controlling at least one of the means selected from the group consisting of an optical magnification means, illumination means, dispensing means for administration of stimulation agents, imaging means, imaging processing means, and/or operator interfacing means.

In another embodiment of the present invention, a method of optical examination of the cervix of a patient is provided, said method comprising the steps of a) applying, at least once, a stimulating and/or contrasting agent to the cervix;

b) acquiring magnified images of the cervix before and after each application of a

stimulation agent;

c) identifying essential anatomical objects in the images, i.e, Os, columnar region and transformation zone; and

d) generating an opacity difference score.

Said patient, is preferably, a mammal, particularly preferred a primate, and more particularly preferred a human. It is important to mention that the above mentioned steps do not necessarily have to be carried out in the given order.

In another preferred embodiment of the present invention, the stimulating and/or contrasting agent is at least one selected from the group consisting of opacity difference score detection agent and/or transformation zone detection agent.

The opacity difference score detection agent has two different purposes: First, essential anatomical objects can be identified after stimulation and/or contrasting with said agent. Second, the opacity difference score detection agent serves to create opacity difference scores, which are indicative of neoplastic or cancerous processes, as the degree of acetowhiteness correlates with higher nuclear density in the respective tissues.

The transformation zone detection agent has, preferably, a deep colour (iodine solution, for example, has a deep purple or brown colour, also termed "mahogony brown"), and serves as a contrasting agent to detect the transformation zone in the cervix, or the area, or shape, of the latter, respectively.

Said opacity difference score detection agent is, preferably, acetic acid, while said transformation zone detection agent is, preferably, iodine solution, such as Lugol's solution or Schiller's solution. Consequently, in yet another preferred embodiment of the method according to the present invention, at least one image, or frame, is acquired before and after application of the opacity difference score detecting agent, (for convenience, these images will be called "pre- and post-acetic acid images" in the following, although said terms will also be used in connection with other opacity difference score detection agents), and/or before and after application of the transformation zone detection agent, (for convenience, these images will be called "pre- and post-iodine solution images" in the following, although said terms will also be used in connection with other transformation zone detection agents).

For the skilled person, alternative stimulating and/or contrasting agents can be derived from the respective literature by routine work, as a matter of course. Alternative agents which can be used to detect areas of higher nuclear density are, for example

Methylene blue. Such alternative stimulating and/or contrasting agents are thus comprised by the scope of protection provided by the present invention. This applies also for the terms "pre- and post-acetic acid images" and/or before "pre- and post-iodine solution images". In case an alternative is used for either acetic acid or iodine solution, these terms will be changed accordingly.

In other preferred embodiments of the method according to the present invention, at least one of the following steps is carried out:

a) In the post-acetic acid image, Os and columnar regions are identified;

b) In the post-iodine solution image, a tentative transformation zone is identified;

c) Said tentative transformation zone is mapped to the pre-acetic acid image and the post-acetic acid image;

d) An actual transformation zone is identified by subtracting, in the pre- and post-acetic acid image, the Os and columnar regions from the tentative transformation zone.

The identification of the actual transformation zone is thus a two step approach: (i) In the first step (step b) of the above list, post-iodine solution images are processed to tentatively detect the transformation zone, based on the color changes that the effected cervical zone depicts on the application of the iodine solution. The post-iodine solution image obtained from the key frame extractor is segmented using color based K- means clustering into two clusters. The smaller of the two clusters is selected. The convex hull of this cluster is defined as the tentative transformation zone; (ii) In a second step (step c) of the above list, the tentative transformation zone is mapped to the pre- and post-acetic acid images and then the detected Os and columnar epithelium regions (detected according to step a) of the above listing, are subtracted to define the actual transformation zone (see step d) of the above listing. The pre- and post-acetic acid images can be registered before identifying the transformation zone. The respective image analysis process is shown in Figs. 6 and 7. In a particularly preferred embodiment of the method according to the present invention, the opacity difference score is generated by image processing of at least one pre- acetic acid image and one post-acetic acid image. For said processing, preferably only the actual transformation zone data of these images are used based on the prior identification of the actual transformation zone.

In another preferred embodiment of the method according to the present invention, the actual transformation zone (or, preferably, its shape and/or overall area), as determined according to the method of the present invention, and/or the opacity difference score as determined according to the method of the present invention, is indicative of a neoplastic and/or cancerous process.

Very often, the actual transformation zone, as determined after iodine solution staining, and mapping the latter to the post-acetic acid image, can also indicate a cancerous region. According to this definition, the opacity difference score as determined by image processing of at least one pre-acetic acid image and one post-acetic acid image serves to determine the degree of whiteness in the cancerous region, and thus the degree, or severity, of the cancer.

In another embodiment, the shape and/or the total area of the actual transformation zone can be indicative of cancer. The opacity difference score can thus be used to confirm, or revoke, the suspicious neoplastic or cancerous region.

It is particularly preferred that the acquisition of magnified images of the cervix, the identification of essential anatomical objects and/or the generation of opacity difference scores is performed with the help of imaging means and/or image processing means.

Preferably, a colposcope is used for this purpose.

It is furthermore preferred that the acquisition of magnified images of the cervix comprises the steps of

^■ Capturing an image stream;

^■ Identifying sequences of interrelated images from said image stream;

^■ Identifying key images from these sequences; and

■ Pre-processing the images.

The identification of sequences of interrelated images and/or the identification of key frames comprises, preferably, the steps of:

^■ Identifying shot boundaries;

^■ Classifying shots; and ^■ Selecting key frames.

Furthermore, it is preferred that the identification of essential anatomical objects in the images comprises the steps of:

^■ Segmenting pre- and post-acetic acid images using K-means clustering of pixels based on their color into 2 clusters;

^■ Labelling smallest cluster as Os + columnar epithelium regions together;

^■ Iteratively removing small and disjoint clusters within both Os and columnar

epithelium regions; and

^■ Separating Os and columnar epithelium using minimum variance quantization.

In a particularly preferred embodiment, the generation of opacity difference score comprises the steps of

Identifying image pixels with dominant opacity changes in the transformation zone of post-acetic acid images,

Mapping said image pixels to the corresponding pixels of the pre-acetic acid images, and

Obtaining the opacity difference score using the following formula:

OpacityDifferenceScore = ^~~~ ( (^ >7) ^_ J( j)) ^{x r} i j))-

It is furthermore preferred that the method according to the invention further comprises at least one step selected from the group consisting of a PAP test and/or a molecular test for the Human Papilloma virus.

The PAP test (named after Dr. George Papanicolaou, who has deveolped it in the first half of the last century) is a screening test used in gynaecology to detect premalignant and malignant processes. A speculum is used to gather cells from the outer opening of the cervix of the uterus and the endocervix. The cells are examined histologically for abnormalities, particularly for neoplastic or cancerous processes of the cervix.

The human papillomavirus (HPV) is a member of the papillomavirus family of viruses that is capable of infecting humans. HPV infection is the cause for nearly all cases of cervical cancer. The presence of viral nucleic acids in a sample taken from the cervix is indicative for HPV infection of the cervix, and is thus a measure for the risk of a patient to develop neoplastic or cancerous processes in the cervix. Modern molecular diagnostic techniques, like PCR, allow for the quick detection of HPV in a sample. One such test is manufactured by Qiagen and is distributed as the digene HC2 HPV DNA test.

The combination of the method according to the invention and one of the two tests set forth above further increases accuracy, sensitivity and reproducibility of cancer detection in the cervix. Furthermore, the use of a system according to the invention for carrying out a method according to the invention is provided.

In one aspect the invention provides for a system for optical examination of the cervix, said system comprising optical magnification means, illumination means, dispensing means for administration of at least one stimulation and/or contrasting agent, imaging means, and image processing means, said image processing means further comprising

^■ key frame extraction means

^■ optionally, glare removal means

■ object detection means, and

^■ opacity change detection means.

In another embodiment the system further comprises operator interfacing means for data input and data output.

In another embodiment the optical magnification means is a colposcope.

In another embodiment said system further comprises a computer workstation for controlling at least one of the means selected from the group consisting of optical magnification means, illumination means, dispensing means for administration of stimulation agents, imaging means, imaging processing means, and/or operator interfacing means.

In another aspect the invention provides for a method of optical examination of the cervix of a patient, said method comprising the steps of a) applying, at least once, a stimulating and/or contrasting agent to the cervix b) acquiring magnified images of the cervix before and after each application of a stimulation agent

c) identifying essential anatomical objects in the images, i.e, Os, columnar region and transformation zone, and

d) generating an opacity difference score.

In another embodiment the stimulating and/or contrasting agent is at least one selected from the group consisting of opacity difference score detection agent, and/or transformation zone detection agent.

In another embodiment the opacity difference score detection agent is acetic acid, and/or the transformation zone detection agent is iodine solution.

In another embodiment at least one image is acquired before and after application of the opacity difference score detecting agent ("pre- and post-acetic acid images") and/or before and after application of the transformation zone detection agent ("pre- and post-iodine solution images").

In another embodiment:

in the post-acetic acid image, Os and columnar regions are identified in the post-iodine solution image, a tentative transformation zone is identified said tentative transformation zone is mapped to the post-acetic acid image, and an actual transformation zone is identified by subtracting, in the pre- and post- acetic acid image, the Os and columnar regions from the tentative transformation zone.

In another embodiment the opacity difference score is generated by image processing of at least one pre-acetic acid image and one post-acetic acid image.

In another embodiment:

a) the actual transformation zone, as determined according to claim 9, and/or b) the opacity difference score as determined according to claim 10, is indicative of a neoplastic, or cancerous, process.

In another embodiment the identification of essential anatomical objects in the images comprises the steps of:

^■ Segmenting pre- and post-acetic acid images using K-means clustering of pixels based on their color into 2 clusters

^■ Labeling smallest cluster as Os + columnar epithelium regions together

^■ Iteratively removing small and disjoint clusters within both Os and columnar

epithelium regions, and

^■ Separating Os and columnar epithelium using minimum variance quantization.

In another embodiment the generation of opacity difference score comprises the steps of

^■ Identifying image pixels with dominant opacity changes in the transformation zone of post-acetic acid images

^■ Mapping said image pixels to the corresponding pixels of the pre-acetic acid images, and

^■ Obtaining the opacity difference score using the following formula Obtaining the opacity difference score using the following formula:

OpacityDifferenceScore = ^~~~ ( (^ >7) ^_ J( j)) ^{x r} i j))-

In another embodiment said method further comprises at least one step selected from the group consisting of a PAP test and/or a molecular test for human Papilloma virus.

In another aspect the invention provides for a system according to -an embodiment of the invention for carrying out a method according to an embodiment of the invention.

The invention provides for a medical imaging system, a computer program product and a method of operating a medical imaging system in the independent claims. Embodiments are given in the dependent claims.

In one aspect the invention provides for a medical imaging system comprising a processor for controlling the medical imaging system. The medical imaging system further comprises a memory for storing machine-executable instructions for execution by the processor. Execution of the instructions causes the processor to receive image data. The image data comprises multiple images of a cervix. The image data may comprise multiple still images or the image data may comprise multiple images which form video data.

Execution of the instructions further causes the processor to select a diagnostic image from the image data.

In another embodiment execution of the instructions further causes the processor to forward the image data to a computer system. That is to say the medical imaging system receives image data, selects a diagnostic image and then forwards the diagnostic image to a computer system.

In another embodiment the image data comprises video data. The video data comprises the multiple images. The video data comprises at least two image sequences. An image sequence as used herein is a collection of images which show the cervix in a particular state. Execution of the instructions causes the processor to identify each of the at least two image sequences. Execution of the instructions further causes the processor to select at least two diagnostic images from each of the at least two image sequences. The diagnostic image is one of the at least two diagnostic images. That is to say the diagnostic image is one of the images selected from an image sequence.

In another embodiment the at least two diagnostic images from each of the at least two image sequences are any one of the following: a single image, a series of images selected at a predetermined interval, and a video clip.

In another embodiment the at least two image sequences are identified by determining a number of feature correspondences of the video data using a feature identification algorithm. An alternative for the analysis of feature correspondences is a method based on color histograms. As used herein the determining of the number of feature correspondences is intended to be equivalent to using the method based on color histograms of the image. The at least two image sequences are further identified by identifying transitions between the image sequence by detecting transitions in the feature

correspondences in the adjacent frames. The transition in the feature correspondences could have several different meanings. For instance it could be a change in the number of feature correspondences identified from frame-to-frame. It could also mean using the strategy of a local minimum to determine the transition. Some embodiments may be identifying transitions between image sequences by analyzing the temporal variation in the number of features over time. One way is by detecting local minimums in the number of feature correspondences measured over a temporal window of more than one frame. In another embodiment the identification of transitions between image sequences is performed by detecting local minimums in the number of identifiable features or the number of corresponding identifiable features in the adjacent frames.

In another embodiment the feature identification algorithm is operable for mapping identical features between the adjacent frames using warping and translation.

In another embodiment the at least two image sequences are any one of the following: a cleaning sequence, a green filter sequence, and an aceto white sequence, an iodine sequence, a detailed regions sequence, and combinations thereof.

In another embodiment the medical imaging system is a system for optical examination of the cervix. Said system comprising optical magnification means, illumination means, dispensing means for administration of at least one simulation and/or contrasting agent, imaging means, and image processing means. The image processing means further comprises key frame extraction means, optionally, glare removal means, object detection means, and opacity change detection means.

In another embodiment execution of the instructions further causes the processor to segment the diagnostic image to generate an image segmentation that identifies the old squamocolumnar junction. Execution of the instructions further causes the processor to determine a quality rating of the diagnostic image using the image segmentation.

Execution of the instructions further causes the processor to reject the diagnostic image if the quality rating is below a predetermined level. Execution of the instructions further causes the processor to accept the diagnostic image if the quality rating is above the predetermined level. This embodiment is advantageous because the medical imaging system can

automatically evaluate if a diagnostic image is of sufficient quality or not. This may enable an unskilled individual to acquire the image data and assign it to a physician or healthcare professional.

In another embodiment the quality rating indicates if the old squamocolumnar junction is completely within the diagnostic image. If a portion of the old squamocolumnar junction is outside of the diagnostic image then the diagnostic image is rejected. In some embodiments if the old squamocolumnar junction is too small within the diagnostic image the image may also be rejected.

In another embodiment the image segmentation further identifies the location of an internal os where the diagnostic image has a center. Execution of the instructions causes the processor to determine a cervix center using the segmentation of an internal os. The cervix center may be the internal os's center. Execution of the instructions further causes the processor to transmit the diagnostic image to place the cervical center at the center of the image. Execution of the instructions further causes the processor to generate a partitioned image which shows the diagnostic image partitioned into a predetermined number of segments. This embodiment may be beneficial because it places the medical image in a form which is more useful for a physician or healthcare provider.

In another embodiment the image is partitioned in 12 pie-like segments.

In another embodiment the image is partitioned into pie-like segments.

In another embodiment the outer boundary of the segmentation is the cervical boundary.

In another embodiment the image is further segmented to locate the: columnar epithelium, transformation zone, and cervical boundary.

In another embodiment the partitioned image is sent to a mobile telephone device of an expert such as a physician or healthcare provider.

In another embodiment the image segmentation further identifies the location of any one of the following: a columnar epithelium, a transformation zone, a cervical boundary, and combinations thereof. Execution of the instructions causes the processor to display at least a portion of the image segmentation in the partitioned image. This

embodiment may be beneficial because the location of any one of the aforementioned regions may help a physician or healthcare provider with performing a diagnosis. It may also enable unskilled or untrained persons to better interpret the partitioned image.

In another embodiment the displayed image segmentation can include: the internal os, a columnar epithelium, a transformation zone, a cervical boundary, the new squamocolumnar junction, and/or the old squamocolumnar junction.

In another embodiment execution of the instructions causes the processor to request a replacement image if the diagnostic image is rejected. This embodiment is advantageous because if an unskilled person is using the medical imaging system he or she can be prompted to acquire another diagnostic image.

In another embodiment the medical imaging system comprises a cloud-based computing system. The cloud-based computing system generates the image segmentation. This embodiment may be advantageous when the image data is acquired using a portable or small handheld optical device. The medical imaging data can be forwarded to a cloud-based computing system for selecting the diagnostic image and performing any image

segmentation. This may have the advantage of providing better healthcare in rural areas where there is a shortage of doctors or other skilled workers. In another embodiment execution of the instructions causes the processor to send the diagnostic image to a mobile-based reporting application. For instance a physician or healthcare provider may have a patient management or reporting application on a smart cell phone, computer or tablet.

In another embodiment the medical imaging system further comprises a camera for acquiring the image data. This camera may be built into the medical imaging system or it may be an external camera which then transfers the image data to the medical imaging system.

In another aspect the invention provides for a computer program for execution by a processor for controlling a medical imaging system. Execution of the instructions causes the processor to receive image data. The image data comprises multiple images of a cervix. Execution of the instructions further causes the processor to select a diagnostic image from the image data. In another aspect the invention provides a method of operating the medical imaging system. The method comprises receiving image data. The image data comprises multiple images of a cervix. The method further comprises selecting a diagnostic image from the diagnostic data.

DEFINITIONS As used herein, the term "optical magnification means" relates to a device, or algorithm, which is capable of magnifying an optical image, for example a magnification lens, a microscope, or a digital image processing system in which the magnification is carried out digitally, or electronically.

As used herein, the term "colposcope" refers to a low-power, stereoscopic, binocular field microscope with a powerful light source, which is used for magnified visual examination of the uterine cervix to help in the diagnosis of cervical cancer.

The term "Os" relates to the external orifice of the cervix, which is an interior narrowing of the uterine cavity. It corresponds to a slight constriction known as the isthmus that can be observed on the surface of the uterus about midway between the apex and base.

The term "columnar region" relates to a region of the epithelium of the cervix.

The ectocervix is composed of non-keratinized stratified squamous epithelium. The endocervix (more proximal, within the uterus) is composed of simple columnar epithelium, i.e., the "columnar region". The term "transformation zone" relates to the area adjacent to the border of the endocervix and ectocervix. The transformation zone undergoes metaplasia numerous times during normal life. This metaplastic potential, however, increases the risk of cancer in this area - the transformation zone is the most common area for cervical cancer to occur.

The term "tentative transformation zone" relates to the transformation zone as provisionally detected after iodine solution staining.

The term "actual transformation zone" relates to the transformation zone as determined after mapping the tentative transformation zone to the post-acetic acid image. The actual transformation zone is often also called "cancerous region" due to high risk of neoplastic, or cancerous, processes.

The term "dispensing means" relates to a device which is useful for applying, in a controlled manner with respect to time, volume and position, at least one stimulation and/or contrasting agent to a given object. Preferably, such dispensing means is a syringe, a pipette or the like.

The term "frame grabber" means a device which has the capability to convert the output of an analogue video frame imaging device or analogue scan converter into a digital image for further image processing.

The term "key frame extraction means" relates to a device or an algorithm that can automatically identify at least one pre-acetic acid image, post-acetic acid image and post- Iodine solution image.

The term "glare removal means" relates to a device, or an algorithm, which is capable of removing glare in a digital image, e.g., as described by Lange et al.

The term "image processing means" (or image processor) relates to a digital image processing device, or software, which is capable of inputting, computing, and outputting digital image data.

The term "opacity change detection means" (or opacity change detector) relates to a digital image processing device, or software, which is capable detecting opacity changes in at least two corresponding images, e.g., as described in the present specification.

The term "object detection means" (or object detector) relates to a digital image processing device, or software, which is capable of detecting and/or identifying objects in a digital image.

A 'computer-readable storage medium' as used herein encompasses any tangible storage medium which may store instructions which are executable by a processor of a computing device. The computer-readable storage medium may be referred to as a computer-readable non-transitory storage medium. The computer-readable storage medium may also be referred to as a tangible computer readable medium. In some embodiments, a computer-readable storage medium may also be able to store data which is able to be accessed by the processor of the computing device. Examples of computer-readable storage media include, but are not limited to: a floppy disk, punched tape, punch cards, a magnetic hard disk drive, a solid state hard disk, flash memory, a USB thumb drive, Random Access Memory (RAM), Read Only Memory (ROM), an optical disk, a magneto-optical disk, and the register file of the processor. Examples of optical disks include Compact Disks (CD) and Digital Versatile Disks (DVD), for example CD-ROM, CD-RW, CD-R, DVD-ROM, DVD- RW, or DVD-R disks. The term computer readable-storage medium also refers to various types of recording media capable of being accessed by the computer device via a network or communication link. For example a data may be retrieved over a modem, over the internet, or over a local area network. References to a computer-readable storage medium should be interpreted as possibly being multiple computer-readable storage mediums. Various executable components of a program or programs may be stored in different locations. The computer-readable storage medium may for instance be multiple computer-readable storage medium within the same computer system. The computer-readable storage medium may also be computer-readable storage medium distributed amongst multiple computer systems or computing devices.

'Computer memory' or 'memory' is an example of a computer-readable storage medium. Computer memory is any memory which is directly accessible to a processor. Examples of computer memory include, but are not limited to: RAM memory, registers, and register files. References to 'computer memory' or 'memory' should be interpreted as possibly being multiple memories. The memory may, for instance, be multiple memories within the same computer system. The memory may also be multiple memories distributed amongst multiple computer systems or computing devices.

'Computer storage' or 'storage' is an example of a computer-readable storage medium. Computer storage is any non- volatile computer-readable storage medium. Examples of computer storage include, but are not limited to: a hard disk drive, a USB thumb drive, a floppy drive, a smart card, a DVD, a CD-ROM, and a solid state hard drive. In some embodiments computer storage may also be computer memory or vice versa. References to 'computer storage' or 'storage' should be interpreted as possibly being multiple storage. The storage may for instance be multiple storage devices within the same computer system or computing device. The storage may also be multiple storages distributed amongst multiple computer systems or computing devices.

A 'processor' as used herein encompasses an electronic component which is able to execute a program or machine executable instruction. References to the computing device comprising "a processor" should be interpreted as possibly containing more than one processor or processing core. The processor may for instance be a multi-core processor. A processor may also refer to a collection of processors within a single computer system or distributed amongst multiple computer systems. The term computing device should also be interpreted to possibly refer to a collection or network of computing devices each comprising a processor or processors. Many programs have their instructions performed by multiple processors that may be within the same computing device or which may even be distributed across multiple computing devices.

A 'user interface' as used herein is an interface which allows a user or operator to interact with a computer or computer system. A 'user interface' may also be referred to as a 'human interface device.' A user interface may provide information or data to the operator and/or receive information or data from the operator. A user interface may enable input from an operator to be received by the computer and may provide output to the user from the computer. In other words, the user interface may allow an operator to control or manipulate a computer and the interface may allow the computer indicate the effects of the operator's control or manipulation. The display of data or information on a display or a graphical user interface is an example of providing information to an operator. The receiving of data through a keyboard, mouse, trackball, touchpad, pointing stick, graphics tablet, joystick, gamepad, webcam, headset, gear sticks, steering wheel, pedals, wired glove, dance pad, remote control, one or more switches, one or more buttons, and accelerometer are all examples of user interface components which enable the receiving of information or data from an operator.

A 'hardware interface' as used herein encompasses a interface which enables the processor of a computer system to interact with and/or control an external computing device and/or apparatus. A hardware interface may allow a processor to send control signals or instructions to an external computing device and/or apparatus. A hardware interface may also enable a processor to exchange data with an external computing device and/or apparatus. Examples of a hardware interface include, but are not limited to: a universal serial bus, IEEE 1394 port, parallel port, IEEE 1284 port, serial port, RS-232 port, IEEE-488 port, Bluetooth connection, Wireless local area network connection, TCP/IP connection, Ethernet connection, control voltage interface, MIDI interface, analog input interface, and digital input interface.

A 'display' or 'display device' as used herein encompasses an output device or a user interface adapted for displaying images or data. A display may output visual, audio, and or tactile data. Examples of a display include, but are not limited to: a computer monitor, a television screen, a touch screen, tactile electronic display, Braille screen, Cathode ray tube (CRT), Storage tube, Bistable display, Electronic paper, Vector display, Flat panel display, Vacuum fluorescent display (VF), Light-emitting diode (LED) displays, Electroluminescent display (ELD), Plasma display panels (PDP), Liquid crystal display (LCD), Organic light- emitting diode displays (OLED), a projector, and Head-mounted display.

BRIEF DESCRIPTION OF THE DRAWINGS

These and other aspects of the invention will be apparent from and elucidated with reference to the embodiments described hereinafter. In the drawings:

Fig. l shows a Philips Goldway Colposcope;

Fig.2 shows a flow diagram of the method according to the invention;

Fig.3 shows a flow diagram for the method to automate colposcope images;

Fig.4a shows a colposcopic image showing the process when acetic acid is applied to the cervix;

Fig.4b shows a raw colposcopic image;

Fig.4c shows the same image with glare pixels removed;

Fig.4d shows the objects in a colposcopic image identified with methods according to the invention, i.e., Cervix (1), Transformation zone (2), Columnar epithelium (3) and Os (4);

Fig.5a shows a colposcopic image taken after acetic acid has been applied;

Fig.5b shows the Os and Columnar epithelium regions identified together; Fig.5c shows the Os and columnar regions separated;

Fig.5d shows the Os and columnar epithelium demarked after removing the small and disjointed clusters;

Fig.6a shows a tentative transformation zone identified in post-iodine solution image (black line);

Fig.6b shows the tentative transformation zone mapped to the post-acetic acid images;

Fig.6c shows the actual transformation zone; Fig.7a shows the transformation zone in post-acetic acid image;

Fig.7b shows the clustering transformation zone to dominant and minor opacity changes;

Fig.7c shows pixels in post-acetic transformation zone with dominant opacity changes;

Fig.7d shows corresponding pixels in pre-acetic acid image;

Fig.8a shows a pre-acetic acid image;

Fig.8b shows a post-acetic acid image after 1 min;

Fig.8c shows a opacity image with opacity difference score = 18.46;

Fig.8d shows a pre-acetic acid image;

Fig.8e shows a post-acetic acid image after 3 min;

Fig.8f shows an opacity image with opacity difference score of 43.28;

Fig. 9 shows a flow diagram which illustrates a method according to an embodiment of the invention;

Fig. 10 shows a flow diagram which illustrates a method according to a further embodiment of the invention;

Fig. 11 shows a flow diagram which illustrates a method according to a further embodiment of the invention;

Fig. 12 illustrates a medical imaging system according to an embodiment of the invention;

Fig. 13 shows a medical imaging system according to an alternative embodiment of the invention;

Fig. 14 shows an example of an unsatisfactory cervicography image;

Fig. 15 shows an example of a satisfactory cervicography image; Fig. 16 illustrates the cervical structure;

Fig. 17 shows two separate workflows;

Fig. 18 illustrates a workflow according to an embodiment of the invention; Fig. 19 shows an example of a medical imaging system according to a further embodiment of the invention;

Fig. 20 shows a segmented cervicography image 2000;

Fig. 21 shows the same image as shown in Fig. 20 divided into 12 pie-like sections;

Fig. 22 illustrates a user-interface which can be implemented in a computer or other computing device such as a smart or mobile phone or a tablet computer;

Fig. 23 shows an acetic acid image where a cervix region is identified;

Fig. 24 shows a Lugol's iodine image wherein a cervix region is identified;

Fig. 25 illustrates the steps in a colposcopic examination;

Fig. 26 illustrates an example of feature-based tracking; and

Fig. 27 shows a plot of the frame number versus the number of tracked features in a sequence of video images.

DETAILED DESCRIPTION OF EMBODIMENTS A colposcope according to the invention (see Fig. 1), has a key frame extractor, pre-processor, object detector and opacity change detector. Key frame extractor takes as input the video that comes from the colposcope which is displayed to the

gynaecologist. The images (pre-acetic acid, post-acetic acid and post-iodine solution) containing the cervical region at the same optical magnification are extracted (in the present application, the words frames and images are used interchangeably). In the pre-processor, glare caused by specular reflection in the images is removed, e.g., by a method described in Lange et al. [, according to which a feature image is extracted from a color image that provides a good glare to background ratio. The glare containing regions in the feature image are detected and extended to cover all pixels that have been affected by the glare. The glare in the affected regions is then removed by filling in an estimate of the underlying image features.

The de-glared image is then sent to the object detector. The object detector processes the images to identify the Os, columnar epithelium and the transformation zone, based on segmentation techniques that use the color and intensity information.

This information is then used by the opacity change detector to estimate the degree of whiteness in the transformation zone and thus the degree of cancer. The flow diagrams of the proposed method are shown in Figs. 2 and 3.

The key frame extractor takes as input the 2D video that comes out of the colposcope. It divides the videos into shots by identifying the shot boundaries and then extracting the relevant images from each shot. The types of images extracted are the pre- acetic acid images, post-acetic acid images and post-iodine solution images having the same magnification factor.

Shot boundary identifier

While performing colpscopy, viewing the cervix at different magnification levels and application of different chemicals, like acetic acid (see Fig. 4a) or iodine solution, to the cervix can be considered as different shots in general video processing applications. A shot/view is defined as a sequence of interrelated images captured from the colposcope, which represents a continuous action in time and space. A "view boundary" is defined herein as the point of switching over from one view to another while performing the colposcopy. Classical shot boundary detection algorithms proposed in context of general purpose video applications can be used to detect shot boundaries. Color features have been used for shot detection.

Key frame selection

Once the shots are identified, key images are automatically selected from each shot. The key frames are from the following shots: pre-acetic acid, post-acetic acid and post- iodine solution. Each frame or image in a shot is described with a set of features based on color, texture, area of the cervix, etc. A score, a is assigned to each frame which is a function of the set of features used. The frame with maximum a is selected as the key frame in a shot.

Preprocessor

The preprocessor takes as input the key images selected by the key frame extractor. Quality of cervical images is attributed to many factors including glare/glint/specular reflection. The glare eliminates the color information in affected pixels and thus makes image analysis algorithms unreliable. It is therefore essential to remove the glare from colposcope images which is done in the preprocessor. The glare regions are detected as small, saturated and high contrast regions. An image, T, is opened in RGB color space. The G component of the RGB color space is used as feature space as it provides a good glare to background ratio. Local maxima for a histogram generated by the G component are identified that represent saturated values. A mask for the glare pixels is generated and applied on the image T to remove the glare pixels from it (see Fig. 4c).

Object Detector

The object detector takes as input the key frame images from which the glare pixels have been filtered out. The object detector aims at identifying relevant objects of the cervix (1), like the Os (4), columnar epithelium (3) and the transformation zone (2), see Fig. 4d.

Os and columnar region detection

Os and columnar regions are identified using the following steps

Segmenting post-acetic colposcope images using K-means clustering of pixels based on their color into 2 clusters;

Smallest cluster is labeled as Os + columnar epithelium region;

• Separate Os and columnar epithelium using minimum variance quantization; • Iteratively remove small and disjoint clusters within both Os and columnar epithelium regions.

The respective image analysis process is shown in Fig. 5.

Transformation zone detection

Detecting transformation zone in the post-acetic acid images is a two step approach. In the first step, post-iodine solution images are processed to tentatively detect the transformation zone based on the color changes that the affected cervical zone depicts on the application of iodine solution. The post-iodine solution image obtained from the key frame extractor is segmented using color based K-means clustering into two clusters. The smaller of the two clusters is selected. The convex hull of this cluster is defined as the tentative transformation zone. This is followed by a second step where the tentative transformation zone is mapped to the pre- and post-acetic acid images and then the detected Os and columnar epithelium regions are subtracted to define the actual transformation zone. The pre- and post-acetic acid images can be registered before identifying the transformation zone. The respective image analysis process is shown in Fig. 6.

Opacity change detector

This module takes as input the transformation zones identified in the pre- and post-acetic acid images and generate an opacity difference score for all post-acetic acid images with respect to the pre-acetic acid images. Broadly, in the post-acetic acid images, pixels in the transformation zone which show dominant opacity changes are identified and they are compared with their corresponding pixels in pre-acetic acid image. The steps to generate the opacity difference score are as follows:

• Convert the RGB values of the transformation zone in post-acetic acid images to Lab color space (L component closely matches human perception of lightness/whiteness);

• Cluster pixels in the transformation zone to two levels of whitish regions, i.e.,

dominant opacity change and minor opacity change (to match opaque white, and translucent white practiced clinically);

• Remove the pixels with minor opacity change;

• Identify the corresponding pixels of the dominant opacity change in the pre-acetic acid image. Finally each post-acetic acid image is given an opacity difference score with respect to the pre-acetic acid image using the following formula:

OpacityDifferenceScore = i j) ^~ J , 7 )) ^{x r} i j))

where

I = Image with dominant pixels in pre-acetic acid image;

j = Image with dominant pixels in post-acetic acid image;

N = Number of pixels with dominant opacity changes;

r = Binary image with dominant opacity changes;

(i,j) = A pixel in the spatial domain of image I / J / r.

The respective image analysis process is shown in Figs. 7 and 8.

Fig. 9 shows a flow diagram which illustrates a method according to an embodiment of the invention. In step 900 image data is received and the image data comprises multiple images of a cervix. Next in step 902 a diagnostic image is selected from the image data.

Fig. 10 shows a flow diagram which illustrates a method according to a further embodiment of the invention. First in step 1000 image data comprising video data is received. The image data comprises multiple images of a cervix. Next in step 1002 image sequences within the video data are identified. Next in step 1004 at least one diagnostic image is selected from each image sequence.

Fig. 11 shows a flow diagram which illustrates a method according to a further embodiment of the invention. First in step 1100 image data is received. The image data contains multiple images of a cervix. Next in step 1102 a diagnostic image is selected from the image data. Next in step 1104 the diagnostic image is segmented to generate an image segmentation that identifies the old squamocolumnar junction. In step 1106 a quality rating is determined of the diagnostic image using the segmentation. Step 1108 is a decision box. The question is is quality rating of the diagnostic image sufficient. If it is yes then the method ends at step 1110. If not the method proceeds to step 1112 and in step 1112 a new diagnostic image is selected from the image data. The method then returns to step 1104 where the new diagnostic image is segmented and the quality is determined again. The method repeats until an image is obtained of sufficient quality. Fig. 12 illustrates a medical imaging system 1200 according to an embodiment of the invention. The medical imaging system 1200 comprises a computer 1202. The computer 1202 has a processor 1204 connected to a user- interface 1206, computer storage 1208, and computer memory 1210. Within the computer storage 1208 there is stored image data 1212 that has been received. It may for example be received from a camera or via a network connection. The computer storage 1208 further comprises a diagnostic image 1214 selected from the image data 1212.

The computer memory 1210 comprises an image selection module 1216. The image selection module 1216 comprises computer executable code which enables the processor 1204 to select the diagnostic image 1214 from the image data 1212.

The computer 1202 could be replaced by a variety of equivalents. The examples include, but are not limited to: a tablet computer, a laptop computer, and a mobile telephone device. The processor 1204 may be represented by various processors distributed amongst various computing devices.

Fig. 13 shows a medical imaging system 1300 according to an alternative embodiment of the invention. The medical imaging system 1300 comprises a computer system 1202 equivalent to the computer system 1202 in Fig. 12. In addition the processor 1204 is connected to a hardware interface 1302. The hardware interface enables the processor 1204 to communicate with a camera 1304.

The computer storage 1208 is shown as containing feature correspondences

1310 calculated from the image data 1212. In this embodiment the image data 1212 comprises video data. The computer storage 1208 further contains image sequences 1312 identified from the video data. The computer storage 1208 further contains an image segmentation 1312 taken from the one or more diagnostic images 1214. The computer storage 1208 further contains a quality rating 1316 of each of the diagnostic images 1214. The computer storage 1208 further contains a cervix center location 1318 of the diagnostic image 1214. The computer storage 1208 further contains a translated diagnostic image 1320 such that the cervix center location 1318 is now in the center of the image. The computer storage 1208 further contains a partitioned image 1322 which is suitable to be displayed to a physician. The partitioned image 1322 may have the cervix center location in the center of the image and is divided into a number of pie-like sections.

The computer memory 1210 is further shown as containing a feature correspondence identification module 1330. The feature correspondence identification module 1330 contains computer-executable code which enables the processor 1204 to calculate the feature correspondences 1310 from the image data 1212. The computer memory 1210 further contains an image sequence identification module 1332 which enables the processor 1204 to determine a set of image sequences 1312 from the image data 1212. Once the image sequences 1312 have been selected the image selection module 1216 selects at least one diagnostic image 1214 for each of the identified image sequences 1312. The computer memory 1210 further contains an image segmentation module 1334. The image segmentation module 1334 contains computer-executable code which enables the processor 1204 to segment the one or more diagnostic images 1214. This generates the image segmentation 1314.

The computer memory 1210 is further shown as containing a quality rating module 1336. The quality rating module 1336 contains computer-executable code which generates a quality rating 1316 for each of the diagnostic images 1214 using the image segmentation 1314. The computer memory 1210 is further shown as containing a cervix center location module 1338. The cervix center location module 1338 contains computer- executable code which enables the processor 1204 to use the image segmentation 1314 to identify the cervix center location 1318. The computer memory 1210 further contains an image processing module 1340. The image processing module 1340 contains computer- executable code which enables the processor 1204 to modify the diagnostic image 1214 into the translated diagnostic image 1330 and then finally into the partitioned image 1322.

Visual inspection is a commonly practiced method for cervical cancer screening especially in resource poor settings. The sensitivity of the procedure increases When this procedure is done under magnification (VIAM). Cervicography is a method for conducting VIAM procedure in which a non-physician takes pictures of the cervix and submits them to a physician for interpretation. One of the main concern of cervicography is that unsatisfactory cervicography should be avoided. An unsatisfactory procedure is where the upper margin of transformation zone in the patient is not visible and yet the images are transferred to the expert for interpretation thus increasing the number of procedures with recall/ revisit of patients. This also results in missed diagnoses.

Cancer arising from cervix is the commonest cancer in women in India. About 30% of cancers in women in India are due to cervical cancer with more than 100,000 new cases diagnosed every year. Unfortunately in India there are only 20 qualified

GynaecoOncologist to screen, diagnose and follow-up 1.3 lakh effected (per year) women in India (only reported cases). Visual inspection (VI) of cervix or visual inspection using acetic acid (VIA) has been effective as a screening tool in resource poor settings. Visual inspection with magnification (VIAM) enhances the sensitivity of the procedure. Both VI and VIAM suffer for high interobserver variation and false positives/ negatives proportional to interpreter experience. In addition VIAM requires specific devices and thus restricts the usage of the procedure. If these limitations can be overcome adoption of this practice can increase and thus in longer term have a positive impact on disease burden and outcome.

On the other hand there is cervicography which has been practiced for some time to address the issue of very low specialist to patient ratio. It is a method for conducting VIAM procedure in which a non physician takes pictures of the cervix and submits them to a physician for interpretation. Cervicography has not been popular due to high false positives and negatives. This largely arises from unsatisfactory or technically uninterpretable cervicographic images for interpretation by experts. Unless the expert can examine the entire transformation zone in its full length, the cervicographic examination is termed inadequate or unsatisfactory (Fig. 14 shows an image of a satisfactory colposcopy and Fig. 15 shows an image of an unsatisfactory colposcopy).

Fig. 14 shows an example of a cervicography image. The cervicography image 1400 shown in Fig. 14 is unsatisfactory and would not be useful for performing a diagnosis.

Fig. 15 also shows a cervicography image 1500. The cervicography image 1500 shown in Fig. 15 is satisfactory and may be used by a physician for performing a diagnosis.

Penetration of mobile phones has been phenomenal in the last few years in countries like India. It has attained the status of an essential device for communication and is used widely, even in rural areas. In order to address the above mentioned limitations of the procedure like VI/VIAM , a mobile based application is proposed in this ID that could make the availability of cervicographic images conveniently in the mobile/ smart phones of specialists and also help the women in getting expert advice on time.

Fig. 16 shows an image which illustrates the cervical structure 1600. The external os is labeled 1602. The columnar epithelium is labeled 1604. The transformation zone is labeled 1606. The original or old squamocolumnar junction is labeled 1608. The new squamocolumnar junction is labeled 1610.

The Cervix is the lower most part of the uterus which extends into the vaginal canal. Ectocervix is the portion of cervix visible through the vaginal canal and is lined by the squamous epithelium. Endocervix is the portion extending from the external Os which is the external orifice of the cervix. This is lined by columnar epithelium. The junction of these two types of epithelium is called the transformation zone. The transformation zone lies between the new squamo columnar junction (New SCJ) and the original squamocolumnar junction (Original SCJ). The transformation zone is the most common area for cervical cancer to occur.

Mobile (smartphones and tablets) health is creating new value: less expensive solutions, new ways to manage care, and better health outcomes and is now a global focus for Philips under the "Mobility Initiative".

Key findings from survey conducted by PWC Healthcare Research Institute (HRI) reveals the following:

Mobile health can improve the use and the value of physicians' time.

One-third of physicians surveyed by HRI said they make decisions based on incomplete information.

They believe the greatest benefit of mobile devices will be to help them make decisions faster as they access more accurate data in real-time.

40% of physicians surveyed said they could eliminate 11% to 30%> of office visits through the use of mobile health technologies like remote monitoring, email, or text messaging with patients. Such shifts could rewrite physician supply and shortage forecasts for the next decade and beyond.

HRI estimates the annual consumer market for remote/mobile monitoring devices to be $7.7 billion to $43 billion. Remote monitoring also could reduce hospital spending, a goal of both government and private payers. Currently, one-third of physicians said they make decisions based on incomplete information for nearly 70% of their patients. Specialists and PCPs find that their biggest obstacle when seeing patients or running their practice is accessing information when and where they need it. Of physicians who are using mobile devices in their practices, 56%> said the devices expedite their decision making and nearly 40% said they decrease administrative time.

The mobile based application for cervicographic images may have the following advantages:

- Requires no special devices like colposcopes but only a mobile phone with camera or mobile phone and a digital camera making the procedure low cost,

- Allows many women to have access to expert opinion thus reducing false

positives/negatives due to lack of experience of interpreter, - Allows only 'satisfactory cervical images' to be sent to the expert thus not wasting their time and decrease false positives/negatives of cervicography, and

- Experts could interpret the images conveniently at their fingertips using their mobile/ smart phones.

Fig. 17 shows two separate workflows 1700, 1702 illustrated as flow diagrams. Workflow 1700 creates the current workflow. Workflow 1702 illustrates the application of an embodiment according to the invention. In both workflows 1700, 1702 a woman is screened for cervical cancer.

Starting with workflow 1700 the woman attends a primary care physician 1704. Next the woman undergoes examination in the form of visual inspection or

cervicography 1706. Next the healthcare worker or general practitioner interprets 1708 the results of the examination performed in step 1706. In step 1710 it is mentioned that false positives are highly likely when step 1708 is performed. Next in step 1712 the woman is referred to a higher center for further cancer screening. And finally, in step 1714 unnecessary referrals adding to the burden on the healthcare system and cost to patient. Workflow 1702 is discussed next. In this workflow starts the step 1720 with the woman attending the primary care physician. Next the woman undergoes examination using VIAM. Next in step 1724 the images are captured under magnification in step 1724. In step 1726 an expert interprets the images captured under magnification. Embodiments of the invention may be used to send images to the expert remotely. Next in step 1728 a referral is performed only if the expert finds the image to be abnormal. In step 1730 the benefit is that this reduces referrals and unnecessary anxiety caused by false positives.

Fig. 18 illustrates a workflow according to an embodiment of the invention. In step 1801 any trained medical professional can capture an image of the cervix at a 5-7x zoom with a digital camera or mobile phone with a macro lens before and after dispensing acetic acid and Lugol's iodine. Next in step 1802 these images are transferred to a hospital server or cloud. In the server or cloud the images are segmented to attain the os, columnar epithelium and cervix region. It is then checked if the cervicography is satisfactory or not. If the cervicography is satisfactory then the cervix region is divided into 12 zones and sent to the expert in step 1803. Next the gynecologist or expert receives a full image of the cervix and the demarked zones one-by-one or the gynecologist could also select the zones which he or she is interested in. These for example could be sent to his or her mobile phone. This is illustrated in step 1804. The expert then annotates and marks the lesions in his or her mobile phone. This is illustrated as step 1805. Next in step 1806 a report is generated containing the annotated images with the diagnosis. The report is then sent back to the hospital server or cloud and then back to the medical professionals as illustrated in step 1806.

Any trained medical professional could capture the image of cervix at 5-7X zoom with a digital camera/ mobile phone with macro lens before and after applying acetic acid and Lugol's Iodine. (Step 1801 as shown in Fig. 18).

These images would be transferred to the hospital server/ cloud (Step 1802 as shown in Fig. 18).

In the server/cloud, the image would be segmented to obtain Os, columnar epithelium and cervix region.

It is checked if the cervicography is satisfactory or not (discussed in the next section).

If the cervicography is satisfactory then the cervix region is divided into 12 zones and sent to the expert (Step 1803 as shown in Fig. 18).

The GynaecoOncologist or the expert receives the full image of the cervix and the demarked zones one by one (or the GynaecoOncologist could select the zones which he/she is interested) in his/her mobile phone (Step 1804 as shown in Fig. 18).

He/She then annotates and marks the lesion in his/her mobile phone (Step 1805 as shown in Fig. 18).

A report is generated containing the annotated images with the diagnosis. The report is then sent back to the hospital server/cloud and then back to the medical

professionals (Step 1806 as shown in Fig. 18).

Embodiments of the invention may have features that make up a system that captures images of the cervix after applying acetic acid and Lugol's Iodine and transfers it to the cloud based system that manages healthcare records. The cloud has an object detection module that segments the cervical images to identify Os, columnar epithelium,

transformation zone and cervical boundary. These objects are input to the adequacy determining module that classifies if the image is adequate for interpretation by an expert or not. If the images are satisfactory then the image is partitioned into 12 regions with the center of the Os as center in the partitioning module. The main image and some (an expert could also select a region interactively)/ all the 12 regions are sent to the mobile phone of the

Gyneco Oncologist for expert advice. The mobile phone has a mCervix module that allows the user, i.e. a Gynecologist or Oncologist to view the cervical images and the twelve regions independently and mark the abnormalities and send it back to the cloud/server. Fig. 19 shows an example of a medical imaging system 1900 according to an embodiment of the invention. The medical imaging system 1900 is illustrated from a functional point of view. First there is an image capture device for the cervix 1902 which provides images to a cloud computing system 1904. Within the cloud computer system 1904 are various software modules for processing the image. First there is an object detection module 1906. Next there is an adequacy determination module 1908 which uses a segmentation generated by the object detection module 1906 to determine if the image is adequate or not. Next the image is processed by a partitioning module 1910 which partitions and/or centers the image. Then finally the cloud computing system 1904 sends the diagnostic image to a mobile and/or smart phone 1912.

The image capture device for the cervix could be a digital camera with a macro lens or a mobile phone with a camera. Images are captured from the cervix after application of acetic acid and Lugol's Iodine. The mobile phone provides an application (browser based) that is used to connect securely to the hospital server/cloud. The images from the phone are transferred to the server with a unique identification of the patient concerned. These images are then analyzed on the server/cloud which are discussed in the next modules.

The object detector module takes as input the images (pre acetic acid/post acetic acid and post Lugol's Iodine) sent to the cloud for a patient objects like the Os, columnar epithelium and the cancerous tissues or transformation zone and cervix region.

Fig. 20 shows a cervicography image 2000. Also shown is an image segmentation represented by various lines. The old squamocolumnar junction is labeled 2002. The new squamocolumnar junction is labeled 2004. The columnar epithelium is labeled 2006. The os boundary is labeled 2008. The os region is labeled 2010. The transformation zone is labeled 2012. The cervix is labeled 2014 and the cervix boundary is labeled 2016.

An adequacy determination module may be used to evaluate the diagnostic image. As discussed earlier, unless the squamocolumnar junction is visible to its full length, a cervicographic image is not said to be satisfactory. This module uses the concept of contour continuity in the neighborhood pixels of the boundary of the transformation zone to decide if the cervicographic image is satisfactory or not. The pseudocode of the algorithm is as follows:

Let: T : the transformation zone in an image I with dimensions A in the x axis and B in the y axis

TB : the boundary of the transformation zone containing pixels (xl,yl) (xn, yn), where x is the X coordinate and y is the Y coordinate of TB in XY plane

C: An array representing the continuity at each point in TB

V(x"i " ,y"i " ) £ TB

If (xi+1 - xi > A/8) & (yi+1 - yi > A/8) then Ci=0 else Ci=l

If n(Ci) : (Ci = 0) >2 then TB is discontinuous else TB is continuous

The object detection module or adequacy determination module could be present in the cloud/server or locally in the mobile phone as an application to give immediate feedback to the medical professional.

As a part of the mobile phone, the adequacy determination module also displays some movie files that could train or guide a medical professional on how to take a cervical image from an unsatisfactory cervicographic image.

Once the cervicography image is considered satisfactory, the image may be partitioned into 12 zones with center of the cervix as center which is shown in the figure below:

Fig. 21 shows the same image 2000 as shown in Fig. 20. However, not all of the segmentations are shown on this image. Only the cervix boundary 2016 and the os boundary 2008 are shown. The region within the cervix boundary 2016 has been divided into 12 pie-like sections 2100. The image has also been manipulated such that a cervix center 2102 has been determined and moved to a central region of the image.

The cervical image as well as each partitioned zone is sent to the expert for his/her opinion. Partitioning the image allows viewing of the image in a mobile phone with small screen easier and come with accurate diagnosis.

In the mCervix module in mobile phones there may be an application that would allow an expert to view images and the partitioned zones in a mobile/smart phone. There may also be an annotation tool. This tool would allow the doctor to view an image, mark the lesions, zoom into some specific part of the image and annotate the image

Report generation tool: This tool would generate a report with essential information. One such template is as illustrated in Fig. 22.

Fig. 22 illustrates a user-interface 2200 which can be implemented in a computer or other computing device such as a smart or mobile phone or a tablet computer. The user-interface shown in 2200 is an annotation tool which may be filled in by a physician which may then be used to generate a diagnosis which is sent to his or her colleagues who originally took the image and sent it to the expert.

Embodiments of the invention may have one or more of the following features:

- A mobile phone based system for cervicography comprising: A mobile phone with camera or a mobile phone and a digital camera; a server or cloud platform for processing the images;

- The mobile phone based system of feature, wherein the mobile phone comprises:

An object detection module and adequacy determination module An mCervix module to annotate and generate a report;

- The mobile phone based system, wherein object detection module is configured for:

Identifying, Os, columnar epithelial regions, transformation zone, etc in pre and post acetic acid images and post Lugol's Iodine images

- The mobile phone based system, wherein the adequacy determination module is configured for:

Identifying if a cervicographic image is satisfactory or unsatisfactory;

Displaying movie files that could guide a medical professional on how to take a cervical image from an unsatisfactory cervicographic image;

- The mobile phone based system, wherein the server/ cloud comprises:

An object detection module and adequacy determination module; and

A partitioning module that partitions the cervical image into 12 zones with center of the Os as the center for partitioning.

The Cervix region may be automatically identified in pre and post acetic acid images using a dynamic thresh holding scheme on the red component of the image opened in RGB color space. This is followed by a K means clustering. If the cluster identifies a region with size less than one/ tenth of the size of the image then we conclude that the cervix is not visible in the image. Else the cluster is considered to be the cervix region. In case of Lugol's Iodine images Otsu thresh holding is used to differentiate the cervix region from non-cervix regions. These algorithms have been verified on 200 pre and post acetic acid images and 30 Lugol's Iodine images and gives accurate segmentation results in 95% cases. The cervix region identified in an acetic acid applied image and a Lugol's Iodine applied image using the above mentioned algorithms illustrated in Figs. 23 and 24.

Fig. 23 shows an acetic acid image 2300. Within the acetic acid image 2300 a cervix region 2302 has been identified. Fig. 24 shows a Lugol's iodine image 2400. Within the Lugol's iodine image 2400 there is a cervix region 2402 identified.

Colposcopic examination may be performed for the diagnosis of cervical cancer. The abnormal area in the cervix is checked for visual signs observed in cervical neoplasia. With a colposcope, a gynecologist captures 5-10 minutes video data. Figure 1 depicts the different phases that can be identified in the colposcopic examination. The video data shows the successive steps of cleaning the cervix, observing the cervix with and without a green color filter, application of acetowhite acid, the temporal response of the tissue to the acid, the application of Lugol's iodine, and detailed (zoomed) analysis of regions of interest of the cervix. Especially the temporal response of the tissue to the acetowhite acid is crucial for the diagnosis. The video sequence is then analyzed in detail by a medical expert.

Interpretation of the colposcopic video data is not trivial and requires years of expertise. Especially in remote rural areas this expertise may not be present. One way to overcome this problem is to send the video data to a medical centre where this expertise is present. An expert can then analyze the video data and send his/her finding back to the remote clinic.

The most relevant markets for colposcopic analysis are India and China. The lack of a stable internet infrastructure, which is especially the case for rural areas, makes it impossible to send large amounts of video data over large distances. Costs are another concern. Even with good infrastructure available, the cost of sending large amounts of data is relatively high.

Colposcopic examination is performed for the diagnosis of cervical cancer. The abnormal area in the cervix is checked for visual signs observed in cervical neoplasia. With a colposcope, a gynecologist captures 5-10 minutes video data. Figure 1 depicts the different phases that can be identified in the colposcopic examination. The video data shows the successive steps of cleaning the cervix, observing the cervix with and without a green color filter, application of acetowhite acid, the temporal response of the tissue to the acid, the application of Lugol's iodine, and detailed (zoomed) analysis of regions of interest of the cervix. Especially the temporal response of the tissue to the acetowhite acid is crucial for the diagnosis. The video sequence is then analyzed in detail by a medical expert.

Interpretation of the colposcopic video data is not trivial and requires years of expertise. Especially in remote rural areas this expertise may not be present. One way to overcome this problem is to send the video data to a medical centre where this expertise is present. An expert can then analyze the video data and send his/her finding back to the remote clinic.

The most relevant markets for colposcopic analysis are India and China. The lack of a stable internet infrastructure, which is especially the case for rural areas, makes it impossible to send large amounts of video data over large distances. Costs are another concern. Even with good infrastructure available, the cost of sending large amounts of data is relatively high.

Fig. 25 illustrates the steps in a colposcopic examination. In step 2502 a normal saline is applied to douche and clean any secretions. Next images may be acquired at this point. Next a green filter 2504 is used and more images may be acquired. The green filter allows the inspection of the blood vessel pattern. Next in step 2506 a 3-5% acetic acid solution is applied. The change in the appearance of the cervix over time and the duration of the change and the time is noted as part of the examination. Next in step 2508 Lugol's iodine is applied and the partial and complete uptake of the iodine is examined. Finally in step 2510 a biopsy is taking from any abnormal tissue regions.

A number of approaches exist for reducing the amount video data. Examples are:

1) Video compression can significantly reduce the amount of data, however at the cost of loss of detail;

2) Video key- frame extraction e.g. used to create a visual summary of a movie by identification of relatively large differences between successive frames.

Add 1. Video compression will not remove irrelevant frames and is only effective if you allow for loss of image detail. For colposcopy, image detail is relevant for at least parts of the recordings (e.g. for detection of the transformation zone or for the analysis of the mosaic pattern of the capillary vasculature).

Add 2. Video key-frame extraction techniques have been developed to provide very effective summaries for movies. The general idea behind key- frames is to capture an image for each part of the video sequence that shows a large difference with the previous part. A strong clue for key- frame extraction is the presence of "black frames" that typically separate different recordings/scenes in a movie. Although the clue of black frames can't be exploited in colposcopy video data, key- frame detection can be used to compress/summarize some parts of the colposcopy video data. However, it is likely to fail in other parts where the image content only changes gradually. One example of small yet important changes of the image data occurs in the part of the video after the acetowhite acid has been applied. From this phase a sufficient amount of images must be sampled in order to allow a proper interpretation of the acetowhite kinetics.

Another example of small yet important changes of the image data occurs during detailed inspection of the cervix (at somewhat higher magnification). From these video fragments at least those images must be collected that, when (spatially) stitched together, cover the complete area scanned in detail with the colposcope.

Therefore, the existing approaches are not applicable for colposcopic applications.

Embodiments of the invention may apply a computer to automatically identify the parts of the video data that are relevant for colposcopic analysis. This algorithm will remove the video data that is not crucial for the expert to come to the right diagnosis. Doing so, the amount of data that needs to be transmitted to the expert centre is significantly reduced. The data that is transmitted may be a limited number of images to capture each different phase of the applied protocol (cleaning, green filter, acetowhite acid, iodine, detailed regions) plus sets of images that were selected using application specific knowledge, e.g. to enable proper interpretation of the acetowhite dynamics.

This approach may:

• significantly reduce the amount of data per patient case that is to be

transmitted to the expert centre

• enable experts to be involved in the diagnosis of cases from rural/remote areas, increasing the quality of the diagnosis

ensure all data relevant for the diagnosis is preserved

reduce the cost for sending data

• significantly reduce of the amount of data per patient that needs to be stored.

Embodiments of the invention may have the following features:

1) Recording of a video image sequence of the cervix using a colposcope;

2) (Semi-) automatic detection of all relevant phases of the recordings;

3) Automatic extraction of at least one image of each relevant phase;

4) Automatic extraction of the sub-set of images that allow for proper

interpretation of the acetowhite kinetics;

5) Automatic extraction of the sub-set of images that allow for detailed inspection of the scan region from video data obtained at higher magnification (zoom modus); 6) Sending the selected images to another location.

In alternative embodiments, one may consider applying compression and/or spatial sub-sampling to one or more of the selected images. This can be done using state-of- the-art image compression techniques or more application specific compression techniques that e.g. more heavily compress parts of the image data outside the cervix. Some of the images allow for strong compression/sub-sampling, others may demand less (or lossless) compression.

Some embodiments of the invention may be constructed by

1) Recording of a video image sequence of the cervix using a colposcope;

2) (Semi-) automatic detection of all relevant phases of the recordings.

This may require information about start and end point of each phase, as well as annotation/labeling of each phase.

A straightforward way to obtain the start and end points in the video for each relevant phase is to ask the user to manually annotate these frames. Automated approaches can identify these start and end locations by studying large differences between successive frames in the video sequence. Another clue can be obtained from monitoring of inter-frame camera motion. Camera motion is typically based on tracking of image features in successive frames. A large number of corresponding features between frames indicates good similarity. When the number of corresponding features drops, it indicates a special event, e.g. occlusion of the colposcope by the user for the purpose of cleaning the cervix or application of acetowhite acid/iodine.

Identification/labeling of each different phase can again be done manually by the user. Automatic approaches will look for image features that characterize each different phase. E.g. a large amount of white image data can be a clue for the cotton used for cleaning the cervix. Pinkish-white color can be a clue for the presence of acetowhite acid and a yellow/orange color can be a strong clue for Lugol's iodine. Information about the protocol, about the duration of each phase, the zoom modus of the colposcope during each phase, and the observed motion pattern, can also be exploited to identify each phase.

3) Automatic extraction of at least one image of each relevant phase.

Some phase can be characterized with only one image. E.g. the cleaning phase or the phase where the cervix is imaged at a fixed magnification using the green filter can be summarized with a single frame. The frame can be selected based on analysis of the motion pattern (small motion indicating small chance of motion artifacts and large chance of proper focus) and/or on inspection of the image content (e.g. check if the Os region is visible and near the image centre).

4) Automatic extraction of the sub-set of images that allow for proper

interpretation of the acetowhite kinetics.

Monitoring of the response of the cervix tissue to the application of the acetowhite acid is a crucial part of the colposcopy examination. Interpretation of the acetowhite kinetics is preferably based on more than 1 or 2 frames. In a most conservative implementation all frames from this phase are considered relevant and transmitted to the expert centre. This part may be compressed using normal video compression techniques. In a more intelligent implementation the dynamics of the changes are accurately monitored and a subset of images is selected that still allows for a proper interpretation.

5) Automatic extraction of the sub-set of images that allow for detailed inspection of the scan region from video data obtained at higher magnification (zoom modus).

To examine the tissue structure in detail the user will zoom- in on the cervix and may scan the region of interest at somewhat larger magnification. This means that neighboring regions will be placed in the (reduced) field of view. By monitoring the motion during this scanning process, and monitoring the quality of the images in the video stream, a subset of good quality images can be selected to (when stitched together) cover the complete scan area.

6) Sending the selected images to another location.

Fig. 26 illustrates an example of feature-based tracking. There is an image 2600 and an image 2602 shown side-by-side. Image 2600 is at time t and image 2602 is at time t+1. Dots connected by a line show corresponding features. The boxes 2604 represent the same location in both images and are used to illustrate motion between frames 2600 and 2602.

Fig. 27 shows a plot of the frame number 2700 versus the number of tracked features 2702 in a sequence of video images. This Fig. shows an example of the correlation between the number of tracked features and the transitions 2704 between different phases of the video. The blocked regions 2704 are regions where the number of tracked features decreases sharply and indicate transitions between image sequences or when an operation was performed on the cervix.

While the invention has been illustrated and described in detail in the drawings and foregoing description, such illustration and description are to be considered illustrative or exemplary and not restrictive; the invention is not limited to the disclosed embodiments. Other variations to the disclosed embodiments can be understood and effected by those skilled in the art in practicing the claimed invention, from a study of the drawings, the disclosure, and the appended claims. In the claims, the word "comprising" does not exclude other elements or steps, and the indefinite article "a" or "an" does not exclude a plurality. The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be used to advantage. Any reference signs in the claims should not be construed as limiting the scope.

LIST OF REFERENCE NUMERALS

1 cervix

2 transformation zone

3 columnar epithelium

4 Os

1200 medical imaging system

1202 computer

1204 processor

1206 user interface

1208 computer storage

1210 computer memory

1212 image data

1214 diagnostic image

1216 image selection module

1300 medical imaging system

1302 hardware interface

1304 camera

1310 feature correspondences

1312 image sequences

1314 image segmentation

1316 quality rating

1318 cervix center location

1320 translated diagnostic image

1322 partitioned image

1330 feature correspondence identification module

1332 image sequence identification module

1334 image segmentation module

1336 quality rating module

1338 cervix center location module

1340 image processing module

1400 cervicography image

1500 cervicography image

1600 cervical structure

1602 external os 1604 columnar epithelium

1606 transformation zone

1608 original or old squamocolumnar junction

1610 new squamocolumnar junction

5 1900 medical imaging system

1902 image capture device

1904 cloud computing system

1906 object detection module

1908 adequacy determination module

10 1910 partitioning module

1912 smart phone

2000 cervicography image

2002 old squamocolumnar junction

2004 new squamocolumar junction

15 2006 columnar epithelium

2008 Os boundary

2010 Os region

2012 transformation zone

2014 cervix

20 2016 cervix boundary

2100 partition

2102 cervix center

2200 user interface

2300 acetic acid image

25 2302 cervix region

2400 Lugol's Iodine image

2402 cervix region

2600 image

2602 image

30 2608 box

2700 frame number

2702 number of tracked features

2704 transition region