This application claims priority from Australian Application No.s 2019901334 and 2019901335 both filed on 17 April 2019, the contents of which are to be taken as incorporated herein by this reference.

TECHNICAL FIELD

This disclosure relates the identification of the molecular level properties of materials using image collection, processing and recognition algorithms.

BACKGROUND ART

Chemical identification is important across a broad range of industries. But identification often involves sending samples to laboratories for detailed analysis. The examples of this can be diverse. For examples when mineral ores are mined it would be useful to have instantaneous information regarding the chemical composition; in cereal crops such as wheat or rice, knowing the protein content is key to ascertaining their commodity value. Typically these types of analyses are performed in laboratories, and the information of the composition of a source arrives long after the sample was sent to its determination location.

Depending on the substance of interest to be identified, different techniques are used to identify chemical composition- they include wet laboratory techniques, x- ray based techniques, microscopy based techniques and spectroscopy based techniques. It often is not practical or not possible to perform these test in the field, as they are expensive and not portable, and involve specialised equipment.

It is to be understood that, if any prior art is referred to herein, such reference does not constitute an admission that the prior art forms a part of the common general knowledge in the art, in Australia or any other country.

SUMMARY

Disclosed is a chemical identification apparatus, comprising: an optical component having a magnification factor, for obtaining a magnified image of a sample; an image acquisition means to acquire an image data from the magnified image; and an image processing module which is adapted to receive the image data, to determine whether at least one of one or more molecular level target signatures is present in the image data.

The molecular level signature is a molecular signature, a morphological signature, a spectral signature, or a combination thereof that can together be interpreted in terms of chemical meaning.

The apparatus can have a light source which is adapted to illuminate the sample, at the time of image acquisition.

The light source consistently illuminates the substance for the acquisition of the image data under an illumination setting. The illumination setting can be chosen to enhance the signature.

The light source can include one or more lighting components. The lighting components can include components which generate lights of different wavelengths.

The lighting components can include infrared light or ultraviolet light.

The image processing module can have an image detection module which is trained to detect said signature.

The image processing module can further comprise a pre-processing module, which pre-processes the image data, and provides the pre-processed data to the image detection module.

The pre-processing module can include one or more pre-processing functions, which are chosen to enhance a possibility of detecting whether the signature is present in the image data.

The optical component can be a microscope.

The apparatus can have a controller which includes said image processing module. The controller can be a processing unit of a computing device.

The computing device can be a mobile device. The optical component can be retrofitted to said mobile device. For example, the optical component can be clipped onto the mobile device.

In a second aspect, the invention provides a method of training an image recognition system having a machine learning algorithm, including providing a plurality of training images to the image recognition system, the training images being magnified images acquired from samples containing a substance having a signature, being a molecular, morphological, or spectral signature, or a combination thereof.

The method includes illuminating a sample of which image data is acquired in said image acquisition. The illumination provided can have a setting which is chosen to enhance the signature.

At least some of the training images which are used to train the detection algorithm can be acquired under an illumination setting which is different than the illumination setting under which the other ones of the training images are acquired. Therefore, training may take place using a plurality of sets of multiple images, each set being taken with different lights and pre-processed prior to training.

The method can include annotating each of the training images to identify portion or portions therein containing the signature.

The training images can include at least one set of positive examples, each set being training images taken of samples known to contain a respective sub-type of the substance.

The method can further include providing negative examples to the image recognition system, the negative examples being at least one set of magnified images taken of samples known to not contain the substance. In particular, the negative examples can include at least one set of magnified images taken of samples known to contain a material which has a signature that resembles the signature of the substance.

The method can include pre-processing the plurality of training images before inputting the one or more training images to the image recognition system. In a third aspect, the invention provides a method of chemical identification or detection, including acquiring an image data of a sample using an apparatus mentioned in the first aspect above. The method can include providing an image data to an image detection system which is trained using the method mentioned in the second aspect above.

In a fourth aspect, the invention provides an application for chemical identification or detection, including an image processing module, the image processing module including an image detection program which is trained using the method mentioned in the second aspect above.

The application can include a control module for a device for controlling an image acquisition device.

The application can include a control module for a light source.

The application can include a user interface module for user to input control commands.

The application can include an executable program which when executed is adapted to cause a display of or associated with a computing device on which the mobile application resides, to display an output.

The output can be an image which is processed by the image processing module, further annotated to indicate location or locations of identified chemicals. In a fifth aspect, the invention provides a computer programme, comprising instructions for controlling a computing device to implement the application mentioned in the fourth aspect mentioned above.

In a sixth aspect, the invention provides a computer readable medium, providing a computer programme mentioned above.

BRIEF DESCRIPTION OF THE DRAWINGS

Embodiments will now be described by way of example only, with reference to the accompanying drawings in which Figure 1 is a schematic depicting the molecular structure of rock salt (NaCl); Figure 2 is a photograph of rock salt as observed under a microscope;

Figure 3 is a photograph of raw rock salt crystals;

Figure 4 is a schematic representation of a device for chemical identification;

Figure 5 is an image of a sample which is photographed to include the sample’s edge portions;

Figure 6 is a schematic representation of a training process for the image recognition program and the image recognition;

Figures 7-1 to 7-4 are images of black ink and green ink, taken under lights of different colours (i.e. wavelengths);

Figures 8-1 to 8-3 are images of a rice grain, taken under red, green, and blue lights, respectively, to show the spectral response of a rice grain; and

Figure 9 is a schematic representation of a portable microscope which can be clipped onto a mobile device.

DETAILED DESCRIPTION

In the following detailed description, reference is made to accompanying drawings which form a part of the detailed description. The illustrative embodiments described in the detailed description, depicted in the drawings and defined in the claims, are not intended to be limiting. Other embodiments may be utilised and other changes may be made without departing from the spirit or scope of the subject matter presented. It will be readily understood that the aspects of the present disclosure, as generally described herein and illustrated in the drawings can be arranged, substituted, combined, separated and designed in a wide variety of different configurations, all of which are contemplated in this disclosure.

The invention discussed herein enables chemical identification using visual images.

Materials at both a macroscopic and microscopic scale materials have visual properties that indicate information about their molecular level structure. Crystalline materials often have their unique structure and morphology. Salt crystals are an example. Figure 1 shows the cubic shape of salt (sodium chloride) at a molecular level. This shape is reflected in the molecular-level crystal packing of sodium chloride. The larger spheres represent the chlorine atoms, and the smaller spheres represent the sodium atoms. Software can distinguish fractured and pristine crystal shapes unique to the cubic lattice type of sodium chloride, and Figures 2 and 3 are photographs which depict the macroscopic morphology which is observable. As can be shown, in these cases, inspection of the material may be sufficient for someone learned in the field to distinguish e.g. sugar crystals from salt crystals because they have different shapes and characteristic morphologies. Both crystalline and non-crystalline materials may also have a unique morphology (characteristic) or other visual properties (particularly, interaction with lights of different wavelengths including infrared light or ultraviolet) which provide visual properties unique to their molecular-level information.

However, the visual properties are hard to quantify without the correct methods to take the images.

When humans examine an image, the properties of colour and shape are simultaneously, rather than separately, observed. Where chemical information is contained in an image, the identification of the chemical requires correlating complex factors which have both“shape” and“colour” components- which are often separated in formal chemical identification.

The present invention provides a device and method for chemical identification, where images are taken in such a way that enables an enhancement of the presentation of their chemical compositional material, or specific molecular or morphological structures, so that they can be used in machine learning of the chemical information. The device thus makes possible the quantification of molecular level chemical identification from visual images, where it is conventionally not feasible to be performed using the human eye, or involves an expensive or cumbersome equipment. Referring to Figure 4, a chemical identification system 100, in accordance with one embodiment of the current invention, includes a camera 102 for acquiring an image of the sample (e.g. cement sheet) containing a substance to be identified. In the example shown in Figure 4, the camera 102 is external to the computing device. However, it can be built into the computing device in alternative implementations.

A magnifying component 112 is coupled to the camera 102. That is, the camera 102 receives with the output of the magnifying component 112. The camera 102 is acquiring a magnified image of the sample.

The system 100 preferably includes a light source for illuminating the sample, at the time of image acquisition by the camera 102. In one example, the light source includes one or more light components, such as light emitting diodes. It can include a combination of coloured lights and white lights. The light source can be built into the computing device, the camera 102, or the magnifying device 112, or it can be a separate device.

In a particular embodiment, a microscope, which provides both the magnifying component and the light source, is used.

The system 100 includes a controller 108, which can be the processor of a computing device such as a computer, a mobile phone, or a tablet. The controller 108 is in wired or wireless communication with the camera 102. For instance, the camera 102 is built into the computing device where the controller 108 resides, then the data transmission is most likely wired.

The detection system 100 can be provided as a mobile application. It can be embodied as an executable program which is adapted to run on the processor of a portable or mobile device such as a laptop, smart phone or a tablet. The detection system 100 can thus be provided in a portable device which the user can bring to the field, to provide on-the-spot substance identification. In an embodiment of this type, the controller 108 includes an application module 110, which when activated, will launch a user interface with which a user interacts to operate the system 100. The user interface may be displayed on a display 114 that is associated with or that is a part of, the portable or mobile device on which the controller 108 is installed. In the interface, the user may be enabled to select a particular class of the chemical or substance they wish to identify, and the selection can be fed to the controller 108 to determine any light setting or image pre-processing which is needed to enhance the visibility of the chemical structure. The light setting is preferably set to the same setting which was used to acquire the training images used for training the detection module for the particular material to be identified.

In an embodiment, the user can manually control the camera 102 and/or the light source 104, by manipulating the camera 102 and/or the light source 104 directly, or by inputting control commands via the controller 108. In the latter case, the controller 108 will include a camera control module and/or a light source control module. However, it is preferred that the light setting is pre-programmed, so that the illumination is automatically controlled by the controller 108, and does not require the user’s manipulation.

The image processing module 106 can be a part of the application module 110. Thus, in an example implementation, the computing device is a mobile device, and the application module 110 is an application which is installed on the mobile device.

The controller 108 includes an image processing module 106 for processing image data from the camera 102. In some example the controller 108 also sends control signals to the camera 102 to operate it, either automatically or on a trigger action by the user (e.g. activating a switch, interacting with a touch screen, voice command, etc).

In some examples, the controller 108 is in data and/or electrical communication to the light source 104 to control the operation of the light source. For example, the controller 108 supplies power to the light source 104, and the power supply is switched by the controller to turn the light source, or one or more light source components, on and off. In other examples, the light source 104 is manually operated, to switch the whole light source, or various lighting components in the light source, on or off. The image processing module 106 will include an image recognition program or module 116 to determine whether the imaged sample contains the substance in question. Before the data is run through the image detection module, it may be pre- processed (such as but not limited to, filtered, colour, contrast or brightness adjusted, transformed by comparison to images from different light sources) by a pre-processing module 118, to accentuate the chemical uniqueness of the material of which the image has been captured. In an example, the controller 108 receives a user input indicating the substance or the class of substance to be identified. The input is passed to the image processing module 106, and image processing module 106 will apply different pre-processing depending on this user input, to enhance the image input so that the chemical signature of the substance is more likely to be detected by the image recognition module 116.

The algorithm for the image recognition module 116 is obtained by using a large set of images to train an image recognition program with appropriate supervised machine learning techniques such as, but not limited to, deep neural networks, to recognise the morphology and/or other structural or spectral properties (e.g. colour) of the substance of interest. Both the structural feature and the spectral feature can be included in the“signature” which is the subject of identification by the image recognition program.

The presentation of the materials depends on the matrix or the“Surrounding Environment”, i.e. how the substance is embedded into its surrounding materials. The detection method utilizes a combination of coloured and or white lights (e.g. LEDs), portable microscopes and image detection, to provide molecular level information about the imaged materials, even if the substance for identification is embedded in a complex matrix.

The image processing module 106 has access to information, which can be built into the system memory or supplied by the user, relating to the substance for identification. The information includes but is not limited to, one or more of morphology information, spectral properties, or colour information characteristic of the substance for identification. It is the features provided by these properties that will be the samples of identification by the image processing module 106.

The method involves training the detection algorithm to recognise the substance. It involves acquiring at least a threshold number of training images, being images of samples known to contain the substance to be identified or detected, i.e., positive examples. The training images may also include negative examples, being images of samples known not to contain the substance of interest. The training images are annotated and provided to the detection algorithm.

Figure 6 depicts a conceptual schematic for training the detection algorithm. The training process 120 used to train an image recognition module 116 includes acquiring a training data set 122. The training data set are training images, taken from samples known to be or known to contain the particular substance of interest. It will be appreciated that the larger the data set, the better the algorithm can be trained. The skilled person will be able to determine, from the application requirements, the number of training images to be acquired. In some embodiments, the training data also include images of samples known not to contain the particular substance of interest, to refine the classification process. In example implementations done during development, between 3000 and 10000 training images were acquired. It will be appreciated that re-training of the system may be performed at any time, as more training images or examples become available.

The image acquisition is adapted to enhance the detection of the chemical or morphological signatures in the image. The training images are acquired with the coupled optical device/lighting - being the magnifying device 112 coupled with the camera 102. This ensures the training images are acquired with a consistent built- in magnification factor compared to the actual sample. In an example implementation (see Figure 8), the magnifying device 112 is a small size microscope with a 60X magnification.

The image acquisition is done under illumination with a predefined setting or one of a plurality of predefined light settings. The light setting includes a setting for the colour of the light (i.e. wavelength) and/or intensity of the light. The precise determination will depend on the substance in question. The settings will be calibrated or chosen to enhance the molecular level signature of the imaged material - e.g. a colour contrast or spectral contrast of the imaged material. The training images can include images acquired with different illumination settings. This helps to train the algorithm to recognise different substances whose chemical information becomes detectable to different extents, when subjected to different light wavelengths or settings (see Figures 7-1 to 7-4). The different light settings also account for situations where different settings may be used to acquire the image to be tested.

In a particular embodiment, during testing, for each test sample a series of test images are taken. Each test image in the series is acquired with a different illumination setting, which is chosen for the recognition of a different signature. The series of test images thus will enable the recognition of the different signatures within the test sample.

Depending on the desired degree of classification, further sets of training images, each including images of samples known to be or include a particular sub-type of the substances of interest, can also be acquired. This enables the detection algorithm to return a“finer grain” result or a finer classification.

Negative training images are also used. Emphasis may be placed on negative examples which resemble the positive examples (i.e. samples or substances which resemble, but are not, the substance of interest). Thus, one or more further sets of training images, each including images of a particular sample, sample class or sample sub-class (depending on the level of classification desired) known to resemble but are not the same as the substance of interest, can also be acquired.

The training images are then annotated, i.e. tagged, to identify areas which show the morphologies of the substance of interest 124. The images, or the tagged or identified areas, are labelled. Depending on the information available, the labelling can provide different levels of information. On the most general level, a yes or no result is return to label the image as showing the substance of interest or not. Or the labelling can provide finer detail to identify positive examples of the particular substance in its different phases or sub-categories, and optionally negative examples of other substances, which are not of interest, with a preference being given to images which shown negative examples which resemble, but are not, the substance of interest.

The classified or labelled training images are fed into an image recognition program 116, which includes a learning component to execute a learning process 126 using the training images. The image recognition program or module 116 will associate each“label” or“class” with the structures that are outlined, highlighted or tagged in the training images of that class or with that label. For instance, the structures can be outlined in boxes or highlighted by an image mask. The training images can be pre-processed prior to being fed to the image recognition program, to facilitate the labelling of the structures/ enhance chemical information imbedded in the image.

The trained image recognition program 116 is included in the image processing module 106, to process the acquired image data 130 (or a pre-processed version thereof) during use, and identify whether the acquired image data show any of the morphologies which the program was trained to identify.

In some embodiments, these training images were processed by different image recognition software programs. Each program can be adapted, e.g. customised, to recognise a different characteristic relating to the signatures. The programs are trained appropriately using sufficient examples of images containing the specific characteristics of interest. In one embodiment, the image recognition programs can each be trained to detect one type of signature. Thus, for a system which is able to identify a number different signatures - e.g. those of various sub-types of the material of interest, or those of different materials - the test image (or a pre- processed version thereof) will be inputted to a corresponding number programs for the identification of the number of signatures. Alternatively, one program can be trained to identify two or more signatures.

Variations and modifications may be made to the parts previously described without departing from the spirit or ambit of the disclosure. As alluded above, some substances have chemical information which can be characteristically identified by how it interacts differently with different colour lights (including IR and UV). For example, Figures 7-1 to 7-4 are images taken with white, red, blue, and green lights respectively, of a sample which includes a line drawn in green ink 130 and a line drawn in black ink 132. The green and black ink 130, 132 each have dyes which reflect red light (see Figure 7-2) to a lesser extent than blue light (see Figure 7-3) and green light (see Figure 7-4). It can be seen that the circled area 134 is less visible under red light (Figure 7-2). By pre-processing the images to emphasize or isolate the spectral properties embedded in the image data, it is possible to optimise the performance of the detection system.

Therefore, in some embodiments, this spectral information is revealed by pre processing the test image, by e.g., passing adding and subtracting images taken with different light sources, to optimise how information characteristic of molecular- level structure can be imbedded in an image. In some embodiments, the training images will include images of the same samples taken with different light settings (e.g. light with different wavelengths).

Figures 8-1 to 8-3 depict another example of a material having different responses to different light spectra. Figures 8-1, 8-2, and 8-3 are images of a rice grain taken under red, green, and blue lights respectively. As clearly demonstrated, the rice grain is more responsive to green light, and more of its structure is captured in the green light image (Figure 8-2). The structure is least responsive to blue light, and the least amount of the details is captured in the blue light image (Figure 8-3).

In a particular embodiment, the system can be retrofitted to, e.g. a user’s existing mobile device having a built-in camera, by loading the program modules needed to the mobile device, and by fitting the magnifying device and light source to the mobile device. For example, a microscope can be permanently or removably retrofitted (e.g. clipped) to the mobile device. Figure 9 schematically depicts a portable microscope 150 having gap 152 which allows the portable microscope 150 to be fitted onto a mobile device (not shown). The attachment can be achieved by a friction fit, a biased clamping or clipping joint, or by another mechanism. When the microscope 150 is attached to the mobile device, the camera lens of the mobile device will line up with the output field of the microscope 150. The microscope 150 includes a light source 154 which as mentioned above, may include different lighting components. The light source 154 is adapted to illuminate the sample 156 of which the test image is being taken. It is preferred that the test images are taken under a consistent light setting, to ensure a consistent performance of the detection system. The light setting can be calibrated, e.g., to supplement pre-processing.

The system described above is applicable to the identification of different chemicals or substances. It provides substantive advantage in the field, as a portable solution which can be used by those who do not work in the areas of chemical identification, to quickly identify substances. As mentioned, this has application in the building and construction industry, or in any application where chemical identification is required or desired - e.g. chemical transportation, mining, excavation, waste identification, etc. As it would be appreciated, in many of these applications, it is impractical to isolate and/or transport the material for spectroscopy or x-ray crystallography analysis at a lab where an expensive apparatus is required, and where the user must wait for the analysis to be concluded.

As alluded above, the various processing functions and control interfaces can be embodied as modules which are provided as executable codes, which are adapted to be installed on the processing unit of the control device.

In the claims which follow and in the preceding description of the invention, except where the context requires otherwise due to express language or necessary implication, the word “comprise” or variations such as “comprises” or “comprising” is used in an inclusive sense, i.e. to specify the presence of the stated features but not to preclude the presence or addition of further features in various embodiments of the invention.