Field of the invention

[0001] The present invention concerns a method, a system and a computer product comprising a hand held device for capturing images of skin portions, in particular for detection of skin lesions and skin cancer, such as melanomas.

Description of related art

[0001] The number of skin cancer has more than double for the last 20 years. This is a cancer that has 95% chance to recover if detected at early stage. Melanoma and skin cancer are the only cancer that can be detected visually and with non invasive methods.

[0002] The ABCD rules (Asymmetry, Border, Colour, Diameter) using naked eyes or macro photography has a low sensitivity (-80%) and a low specificity (-75%) depending of the various studies done on this subject. Also this method is even much harder to use for the very early stage detection. For early stage melanoma ABCD gives only -50% sensitivity and -60% specificity.

[0003] Dermatoscopy is a rather a recent method (really used at large scale only after the year 2000) used to detect skin cancer at early stage. This method can reach a sensitivity of detection of > 95 %in sensitivity with a well trained dermatologist. This method is used more and more often by the specialist but requires some good training to be efficient. But the sensitivity level of detection is such higher versus the other method that this should be much broadly used.

[0004] Today, electronic dermatoscopes are targeting the professional market and would be in average very expensive for a non professional. [0005] Thanks to the new technologies of information (tele-video, web etc) that have so much improved for the last 10 years that tele-medicine starts to be a real subject of actuality. However the implementation of such system tends to be really not as common as it could be expected. Many doctors are reticent even though there is a clear market need from the patient.

[0006] For the detection of skin cancer there are today few solutions in the market to get a first level of information to get a skin lesion controlled, but usually it is only by using standard macro photography (with a smartphone, a digital camera, i.e.). Even if this can be used for an auto control and observe variation of the lesion over time, this is definitively not sensitive enough to have a secure pre diagnostic based on a picture only. Those solutions are definitively not suitable for regular and full screening check having many spots to control.

[0007] One way to improve the quality of the images and the quantity of information that can be retrieved from each image is to spread o gel or liquid on the captured skin portion. For example it is known to apply a gel, such as a dermoscopic gels or liquids (water, alcohol 70%, etc) , that will reveal structures under the surface of the skin that won't be visible without the gel. This technology is known as epiluminescence microscopy. However, the gel or liquid must be carefully spread over the skin, in an adequate height, and without any bubbles. This is difficult to achieve for non specialists, and especiallytime-consuming even for specialists, especially when a large number of moles or skin lesions need to be examined. If the amount of gel or liquid varies, the quality of the images decreases, and a comparison between successive images of the same skin portion is difficult.

[0008] Another technique is based on the use of polarized light to achieve similar type of images. Both techniques can provide complementary information, but the trend and according to some studies is to use wet dermoscopy to cover more type of pattern recognition while and better for final diagnosis, while polarized can be enough for first level of screening. [0009] US2004092802 discloses a hand held device and a method for determining the variation in concentration of chromophores within an epithelial tissue. The proposed device comprises lights sources, image sensor, data processing and displaying means. The device could comprise a patient-contacting film that iscoated with index matching oil.

Maindrawbacks of this solution derive from the limited quantity of index matching oil that could be coated on the film. Actually, this quantity limits the number of manipulations on the patient's skin during both the preparation and the images acquisition phases. [0010] US20120172685 discloses a skin analysis system.This system is in form of additional equipment for consumer electronic devices. The proposed solution allows periodically image acquisitionsof moles, images comparisons andcalculation of moles development rates.The device could generate alerts, inviting user to an urging consultation to medical specialists. The systemprovidesconsumables in form of a sheet with self- adhesive elements to enter information in the image, in particular ID numbers, dates, names andimage calibration elements.These consumables are to bestuck upon the skin around the region of interest. The location of the skin lesions images is provided by taking a large body portion image where multiple skin lesions are captured together with said consumables.

[0011] WO02080771 discloses a disposable calibrated camera for skin lesion examination. This camera assembly is equipped with a transparent window that, during the use, could be coated with an index matching liquid. In one embodiment, a liquid reservoir equips the camera assembly. However, this camera assembly focuses on a single-use mode. Actually, camera assembly components are specifically designed for a single use, limiting the employ of skin anomalies evolutions analysis and survey methods that are indispensable to support skin lesions and melanomas detections. [0012] It is an aim of the present invention to provide a new and improved method and a new and improved device for the detection of skin lesions and melanomas. Brief summary of the invention

[0013] According to the invention, these aims are achieved by means of methods, systems and computer products comprising the claimed features.

[0014] In one aspect, the invention is related to a method for capturing an image of a skin portion for dermatoscopy analysis, comprising the following steps:

placing an hand held device against a skin portion;

capturing an image of the skin portion with an image sensor through a lensof the hand held device;

storing said image into a memory of said hand held device;

automatically determining a body area from which said skin portion belongs.

[0015] According to the invention, the method includes a step of automatically determining a body area from which the acquired image is or will be captured. The purpose is to code the picture with the body area to make easier immediate software treatment and/or offline software treatment at later stage.

[0016] In one embodiment, a skin portion which is captured for the first time is manually identified by the user, for example by selecting a body area in a menu, by typing or speaking a body area indication, by scanning a corresponding code-bar or QR code etc. If the same skin portion is captured a second time, and all subsequent times, the system might recognise this skin portion, using for example computer vision algorithms and/or displacements detected with an inertial sensor, and automatically retrieve the previously entered identification of body area.

[0017] In one embodiment, the method includes an operation of automatically matching the acquired image with a previously captured image in order to, automatically or manually, determining the body area to which said previously captured image belongs. This operation could compriseretrieving a signature from the captured image and comparing this signature with already available signatures associated with various body areas. The signature could, for example, depend on skin natural markers such as nevus, skin pores, skin lesions, skin grains, hairs, buttons and/or veins. This unique digital image signature will be used to compare future images of the same skin portion in order to make a secure time series comparison of one identified lesion.

[0018] In another embodiment, the method includes a step of

automatically determining a body area from which the acquired image is or will be capturedthat further builds an image of a larger body area by stitching the acquired image other images.

[0019] In another embodiment, the method includes a step of

automatically determining a body area from witch the acquired image is or will be captured that further uses an inertial sensor in the hand-held device for determining the position of said hand-held device. [0020] In one embodiment, the method includes further comprises a step of user selecting the body area from which the image is or will be captured among a list of predefined body areas. This operation could be made by manipulating a hand held device interface, as a roller, or by capturing an image of a code-bar associated with a particular body area, or other type of visual identification (QR code etc). The selection could be supported by a hand held device display.

[0021] In another aspect, the invention is related to a system comprising a hand-held device for capturing an image of a skin portion for

dermatoscopy analysis, comprising:

an image sensorfor capturing said image;

a lens arranged for projecting a focused image of the skin on the sensor when the device is in direct contact with the skin;

a memory for storing the captured digital image;

body area determination means for automatically determining a body area from which the skin portion belongs. [0022] In one embodiment, the system further comprises means for automatically matching said image with a previously captured image, and automatically or manually determining the body area to which said previously captured image belongs. [0023] In one embodiment, the system further comprisesmeans for retrieving a signature from the captured image and comparing this signature with already available signatures that are associated with various body areas.

[0024] In another embodiment, the system further comprisesmeans for stitching the acquired image with other images to build an image of a larger body area.

[0025] In one embodiment, the system further comprises inertial sensor in the hand-held device for determining the position of the hand held device. [0026] In one embodiment, the systemfurther comprises data selection means for indicating the body area from which an image is taken. The data selection means could be arranged for allowing a user selection of one body area among predefined body areas. The hand held device could be so equipped with a roller, a display and/or with a code bar reader. [0027] In another embodiment, the system further comprises external apparatus that is connected with the hand held device and that comprises at least one or at least one part of the body area determination means.

[0028] In another embodiment, the device further comprises software means for including an indication of body area as metadata in the corresponding captured image.

[0029] In another embodiment, the device further comprises software means for automatic identification of image area containing skin lesions. [0030] In another aspect, the invention is related to a computer product computer code to be executed by a microprocessor in one of the above mentioned systems for causing the microprocessor to carry out the steps of one of the above mentioned methods, when this microprocessor executes this computer code.

Brief Description of the Drawings

[0031] The invention will be better understood with the aid of the description of an embodiment given by way of example and illustrated by the figures, in which: Fig. 1 shows a view over anhand held device.

Fig. 2 shows markers on the skin contact glass.

Fig. 3 shows a single use contact glass with the contact gel already on the contact glass.

Fig. 4 shows a hand held device with contact gel and/or liquid tankaccording to the invention.

Fig. 5 shows details of a hand held device contact glass with contact gel and/or liquid distribution canal or groove according to the invention.

Fig. 6 shows a device equipped with a contact glass provided by contact gel already on the contact glass.

Fig. 7 shows a hand held device with a light pointer.

Fig. 8 shows a view of a spot with markers clearly visible.

Fig. 9 shows successive views of the same spot taken at different time. Fig. 10 shows two successive pictures of a same spot where similar areas are circled.

Fig. 1 1 shows an example of a reference picture of a skin portion used for automatic or manual determination of skin type. Fig. 12 shows a skin image capturing system comprising a hand held device with contact gel and/or liquid tank and a connected computer.

Fig. 13 shows a skin image capturing system comprising a hand held device and a computer. Detailed Description of possible embodiments of the Invention

[0032] Figure 1 shows a device, or dermoscope, comprising an image sensor and a contact glass 2 that will be in contact with the skin for dermatoscopy picture. It is also possible to have a device with a lens or glass maintained at a predefined distance of the skin but without direct contact. The captured images of skin portions can be associated with metadata which can be used by a software in a remote computer 66 for analysing the pictures as showed in Figs.12-13.

[0033] The focus of the device is set at factory so that a lens 70 projects a focused image of the skin portion on the image sensor. The user has not to worry about adjusting the focus.

[0034] The skin contact glass 2 may have some markers 6 on it (Fig. 2) that will be used for some software treatment. There might be some white and black reference dots. There might be some reference colour dots.

[0035] The markers are clearly visible on each image captured with the device, as shown on Figs. 8-1 1 . The colour and/or brightness of those reference dots are known and might be used for any image treatment during inspection by the user or the dermatologist, for calibrating the colour and brightness and saturation of the stored and captured image, and/or the colour of the displayed and/or printed image. The expected colour and brightness of the reference marks can be compared with their colours in the captured images, to adjust the colour and brightness of the whole image.

[0036] Furthermore, the orientation of successive images of a same spot, for example successive views taken at different times can be adjustedby software, by comparing matching patterns in different orientations, as shown on Figure 9. Asoftwaremodule in the device 1 , or in a computer 66 receiving the images captured with the device, can be used for adapting the orientation of various images of a same portion of skin and facilitating their comparison.

[0037] This is a way to calibrate the colours and to have systematically the same results independently to the screen or computer platform used. This allows making sure that comparing pictures of the same lesion at different time, a reference is available for a true comparison in order to detect correctly any colour changes.

[0038] The software used to visualize the lesion will automatically adjust the colour and brightness based on the reference dot. This might also be used for automatic or human detection of significant patterns.

[0039] The device 1 may include a single use contact glass 2 with the contact gel and or liquid 8 already on the glass. An example of single-use glass 2 with gel 8 on it is illustrated on Figure 3. In the example, a thin protection foil 1 1 is used to protect this contact gel and/or liquid layer 8 during transportation and storage of this contact glass in a packaging blister 10. The contact glass can be removed from the blister and mounted at the extremity of the device 1 , as illustrated on Figures 6, 12 and 13. The contact glass with the gel on it may be replaced after each image, or after each series of images captured from a patient. If the amount of liquid or gel on the contact glass is not sufficient, it might be possible to add more gel from a tank 22 comprised in the device. [0040] Alternatively, the device 1 might comprise a contact glass that will be used substantially during the whole life of the device 1 , without need to be replaced after each use.

[0041] In both embodiments, some gel or water needs to be applied on the skin for every new spot to check. In order to make the process faster, easy to use, and more reproducible, the device is equipped with a tank 22 for storing a plurality of portions of contact gel and/or liquid as shown on Fig. 4. When the device 20 is on contact with the skin, this tank could be actuated in order to dispense the right quantity of gel and/or water through a distribution canal 23 equipped with an aperture to make the dermatoscopy picture.

[0042] This tank 22 could be located on the external surface of the device body 5 in order to be manipulated by the device's user. The tank could be, for example, entirely or partially made of elastic materials. A simple pressure of the tank with at least one finger will distribute the required portion of contact gel and/or liquid from the tank on the skin portion.

[0043] Alternatively, the device could comprise a lever (not shown) whose movement actuates in a mechanical way the tank, delivering a more precisely defined portion of contact gel and/or liquid onto the skin and/or onto the contact glass.

[0044] Alternatively, the device could comprisean electro-mechanical actuator activated by a switch, a button and/or a roller to deliver a precise portion of gel or liquid onto the skin and/or onto the contact glass. [0045] The tank could be located within the device body 5.

[0046] A part of distribution canal 23 could be located on or within the hand held device body 5. [0047] A part of the distribution canal 23 could be located on or within the hand held device extender 3.

[0048] The tank may be provided with an overture and a lid for contact gel and/or liquid recharge purpose. [0049] In case of need, the tank could be made of a disposable contact gel and/or liquid cartridge for a safe, clean and rapid recharge.

[0050] The device could be equipped with distribution nozzle 21 to control the direction, the flow and the quantity of contact gel and/or liquid that is to spread on the skin and/or on the contact glass 2. [0051] Various types of nozzle could be employed, as for example, jet nozzle, spray nozzle and flat stream nozzle. Multiple nozzles of the same type or combination of various types could be used to improve the efficacy and regularity of the spreading operation.

[0052] In one embodiment, the gel or the liquid is applied first on the skin or on the contact glass of the dermatoscope, and then the device is applied and pressed on the lesion. This requires some practice and fast handling operation when the gel or liquid is applied on vertical skin portion.

[0053] Therefore, in another embodiment, a small canal is provided in the glass, or a groove 24 along the body of the device, that will help to bring the gel or liquid to the lesion 33 (Fig. 5). In such case, the device is pressed on the skin lesion and then we push the liquid through. This embodiment is also more comfortable for the patient because of less liquid spreading and flowing all over. [0054] Using wet dermoscopy requires to apply the right pressure. If the pressure is too low, the epiluminescence is not appearing and more air bubbles might be seen. If the pressure is too high, the liquid might be faster flowing outside the area. If the pressure is really above a normal and reasonable pressure it can influence the blood vessel close by and change some aspect ofthe lesion. But usually after few shots only, it is easy to get it right. Nevertheless, one feature of the invention is a small pressure sensor 69 that will indicate to the user by a beep and / or small led and / or other visual indicator that the pressure is right or too high or too low.

[0055] Those indications can also be used in combination with an interactive web base platform to drive the user to the best and optimum image capture process.

[0056] The small pressure sensor could be mounted in order to measure the pressure between the contact glass 2 and the extender 3, or

alternatively, between the extender 3 and the body 5 of the hand held device.

[0057] The device may comprise a light pointer 31 shown on Figure7 that projects a focused beam on the skin 30 where the centre of the image is.This help to centre small lesions 32 of few millimetres diameter only. The ray of light pointer can be with LED or laser.The power must be very low and should not risk any injuries to the eyes or the skin.

[0058] Just before the snap shot, the device 1 automatically switches off the ray of light pointer 32 and turns on awhite LED, or a plurality of LEDS producing in combination a white light.The LED remains on only few seconds.The power of the LED is automatically adjusted to have the reference spots on the skin contact glass correctly lighting.

[0059] The device further comprises indication retrieving means of body area from which the acquired image is or will be captured. The purpose is to code the picture with the body area to make easier some offline software treatment at later stage.

[0060] This means could be automatically executed and/or manually driven. [0061] The indication retrieving means could comprise means for automatically matching the captured image with a previously captured image stored in the memory 68, and automatically or manually

determining the body area to which said previously captured image belongs. A signature from the captured image could be retrieved and compared with signatures already available and associated with a body area. This signature could depend on skin natural markers such as nevus 36, skin pores, skin lesions 33, skin grains, hairs 34, buttons and/or veins (Fig. 9).

[0062] The indication retrieving means could further comprise means for stitching the acquired image with other stored images, and building an image of a larger body area.

[0063] The indication retrieving means could further comprise an inertial sensor in the hand-held device for determining the position of said hand-held device. [0064] Before to start shooting on some skin lesion, the patient could furthermore select manually the body area onwhich he or she has the lesion to be controlled.

[0065] An example of device with a menu based selection, body area selection roller 4 and display 40 is shown on Figures 4, 6, 7, 12 and 13. [0066] The device could further comprise a code bar reader exploiting the device sensor 61 . The body area could so manually selected capturing a code bar image indicating a particular body area. The code-bar could be a conventional code-bar, a QR-code or any other type of visual coding that can be used as identification with image analyzes. [0067] The body area can be identified such as right forearm, left forearm, neck, top hand right, etc. [0068] With detailed body map and multiple code bar or QR code or other, the localisation can be much more precisely identified (for instance with a number, such as for example ten, different zone on the left arm)

[0069] All the pictures captured are saved on an onboard memory 68, which could be a FRAM, and/or a removable SD cards for instance. Pictures stored in the memory 68 may be downloaded to a computer when the device 1 or the portable memory (SD cards) is plugged into a computer 66.A USB interface 63 could be used for this purpose. Pictures can also be transferred over a wireless interface, for example over wifi or Bluetooth or any wireless technology.

[0070] The device contains at least one battery 64 that is rechargeable via USB port 63 or via a battery loader or loading station.

[0071] The device contains a real time clock 67 that allows to clearly identifying the time of capture of each picture. [0072] The device 1 contains all the necessary code to automatically open and connect to the web application, in order to transfer the images to a remote server for storage, comparison with previously captured images of the same skin portion, and/or analysis by a dermatologist or

automatically with a computer vision system for example. [0073] A coding of each picture is used throughout the web application. The device associates a unique code similar to what is described below to the picture:

ID: 1 12395657824 - 001 - 00002 - 0000001 - 2501 122102 - xxxxxx - xxxxxx - xxxxxx (Device - user - location - Picture number - date / time picture - flag diagnose - date diagnose - Visa). [0074] This code could be transferred along with the picture to a remote computer 66 or server where is can be used for classifying and retrieving the picture, and for facilitating the comparison with other images of the same spot at different times. The user identification part of the code can be an anonymous identification or an alias.

[0075] The images captured with the device can be processed locally in the device, for example in order to adjust their colour, contrast and brightness, possibly based on the reference marks. Moreover, a software module in the device might carry out computer vision steps for

automatically identifying images areas which might with some probability match a skin cancer or another skin lesion. An indication might be displayed to the user, for example a visual or audible indication, when a suspicious spot has been found on the skin.

[0076] The images might then be transferred to a further computer, for example over a wireless interface, a USB cable, a memory card and so on. The images might also be transferred to a remote computer over an

Internet connection. The computer receives the images with associated metadata, such as a picture code, and carries out further methods for processing the images, classifying them and for assisting the user or the dermatologist in detecting skin lesions and melanomas and/or automatic detection of suspicious images.

[0077] The device can be used with a real time interactive web base application for guidance, real time information and training and feedback.

[0078] The software in the device and/or in the computer might carry out an automatic detection of skin cancer with digital image analysis. The software might also assist the user or dermatologist and help him in the fast and reliable detection of melanomas. The specialist operator while checking and scrolling through hundreds of pictures for a pre diagnostics, is thus driven through all the steps with software assistance to highlight some area or characteristics when required, and to help him in verifying the images and detecting more quickly and in a more reliable way the most critical images.

[0079] For example, the software executed on the computer can detect that several pictures taken at different time correspond to a same spot. The comparison might be used on the date - it is unlikely that several pictures of same spot taken on the same day need to be compared -, on the body part identification, and on computer vision algorithms for matching successive views of a same spot when those views include similar group of pixels, or possible transformation from one view to the next. Fig. 10 shows two views of a same spot taken at two months intervals; the circled area 35 corresponds to similar group of pixels which can be used for automatic matching of views.

[0080] The software can also be used for facilitating manual matching of successive images, for example by presenting to the user a selection of images likely to correspond and that the user can manually match.

[0081] The software can also be used for automatic detection of areas that have changed between successive pictures, and automatic marking of those areas, or automatic marking of series of pictures in which significant changes were observed. [0082] The software might detect patterns or image elements

representative of different types of lesions. Skin cancer such as melanomas can have various levels of the development characterized by changing aspects and various patterns. The software might comprise routines or modules for detecting image portions representative of some of those levels. For example, a module can be executed for detecting and

highlighting pictures having blue gray globules which are sometimes hard to see but very specific to carcinoma at level 2. The software can highlight the pictures having those areas, and in those pictures highlight the area that matches the characteristics of the blue gray globules. At level 6, the software might highlight the group of pixels that have similar colour of blood vessels and organized like blood vessel. [0083] The process might also comprise a step of indicating of detecting the skin type of the patient. A skin type might be associated to a series of pictures taken during a session. The skin type might be included in the image coding. The skin type might be determined from a picture taken at a specific location, for example a location without melanomas. An example of image which can be used for automatic and/or manual determination of skin type is shown on Figure 10. The skin type might be indicated as a skin index type and might depend on the skin colour.

List of elements used in the drawings

1 : Hand held device for capturing images of skin portion

2: Contact glass

3: Extender

4: Roller for data selection

5: Body of the hand held device

6: Image calibration marker

7: Acquiring button

8: Contact gel and/or liquid

9: Field-of-view of the image sensor

10: Contact glass packaging blister

1 1 : Thin protection foil

20: Hand held device for capturing images of skin portion

21 : Contact gel and/or liquid distribution nozzle

22: Contact gel and/or liquid tank

23: Contact gel and/or liquid distribution canal

24: Contact gel and/or liquid distribution canal or groove within the contact glass

30: Skin surface

31 : Ray light focusing point

32: Skin portion of interest

33: Skin anomalies

34: Hair

35: Similar areas between two successive pictures of a same spot

36: Nevus

40: LCD display

60: Lighting system

61 : Image sensor

62: Microprocessor

63: USBinterface

64: Battery pack

65: USB cable : Personal Computer: Real time clock: Memory

: Pressure sensor: Lens