This application claims the benefit of U.S. Provisional Application No. 61/809,429, filed 8 April, 2013.

Field of the Invention

The present invention relates to the field of automated quality assurance, sample inspection, and absolute color determination of samples such as grains, pulses, spices, powders, medical cannabis, liquids, bolts, printed circuits, and the like.

Background of the Invention

Optical devices for quality assurance are common for inspection and quality control, where special precautions are taken to assure necessary conditions of lighting, shadowing and the like. However as will be appreciated, different setups will often have slightly different conditions (of ambient lighting and the like) and hence direct comparison of analyses performed at two different sites (even for production of the same product) may not be possible.

There is therefore a long-felt need for a device capable of guaranteeing exact conditions of lighting and hence direct comparison of analyses performed at different locations.

There are also in many cases requirements for checking a color of a sample or its parts, which are met in primitive fashion by employing a spectrophotometer to analyze a single point or small group of points. This method has the drawbacks that noise may severely affect the statistical accuracy of a small number of datapoints, and that furthermore spatial color patterns cannot be checked.

Thus there is also a longfelt need for a device capable of determining the spatial spectral distribution of a sample, suitable for use in the production of textiles, grains, pulses, spices, and the like, able to replace a human inspector, and performing faster tests with more accurate and repeatable results, as will be explained further. Summary of the Invention

The invention comprises a method for automatically checking samples in terms of correctness of color, shape, size, and other parameters using image processing technology. 'Correctness' here is defined in comparison to an instance called the Master sample which is used to define the desired characteristics.

The invention comprises a combination of hardware and data processing methods adapted to produce 'go/no-go' assessments, and discrimination into classes of samples of materials such as: grains, seeds, flours, processed materials, spices, mechanical parts, printed circuits, and the like.

The invention also comprises a device adapted to perform the above operations.

The hardware comprises a closed box having a hemispherical reflection chamber, described in provisional patent application US61/809429 which is incorporated herein by reference. The hardware is referred to in the figures to follow, and the data processing methods employed use routines described by the figures as well.

The device consists of a portable, fully enclosed sample analysis chamber such as that shown in Fig. la,b. The chamber has two halves which may be opened (Fig. lb) and closed (Fig. la) to allow insertion of a sample and then closing of the device to keep out ambient light. The upper half has a downward-facing hemispherical dome, attached by hinges to the lower half which supports a planar analysis surface. The dome is provided with lighting means such as RGB LEDs disposed so as to provide uniform illumination of controllable color and intensity to the planar analysis surface when the device is closed.

The device is opened and a sample of material (which may be powder, granular, bulk, or the like) is placed on the analysis surface (or on a jig placed on the surface). The device is then closed and the sample is subsequently analyzed, automatically or manually, by means of a calibrated camera at the top of the dome. described in the detailed description to follow, including a top LED ring, structured light illumination, and more.

The foregoing embodiments of the invention have been described and illustrated in conjunction with systems and methods thereof, which are meant to be merely illustrative, and not limiting. Furthermore just as every particular reference may embody particular methods/sy stems, yet not require such, ultimately such teaching is meant for all expressions notwithstanding the use of particular embodiments.

Brief Description of the Drawings

Embodiments and features of the present invention are described herein in conjunction with the following drawings:

Figs. 1A,B shows side views of the inventive device in closed and opened configurations.

Fig. 2 shows a side view of the top half of the device. Fig. 3 shows a side view of the device.

Fig. 4A shows details of the device assembly including sample stage. Fig. 4B shows details of the device assembly including gain correction plate. Figs. 4C,D illustrate the multiple lighting sources of the invention. Fig. 5A shows a sample stage having grasping means.

Figs. 5B,C shows a sample stage having target plate positioning means and sample positioning means.

Figs. 5D,E shows a sample stage with a homography calibration plate in place.

Figs. 6A,B,C presents a vibration stage.

Figs. 7A,B,C show means of producing structured light.

Fig. 8 illustrates a human operator near a machine of the invention.

Fig. 9 presents a flowchart for image analysis using the system. of the system.

Fig. 11 shows one exemplary embodiment of the invention.

Fig. 12 shows a result of rice analysis screenshot, where some of the rice is found broken.

Fig. 13 is a screenshot of parsley analysis, where the areas with dark or bright exceptional color are marked. The total amount of relative area with exceptional color in the sample is high, and therefore the sample is classified as a reject.

It should be understood that the drawings are not necessarily drawn to scale.

Detailed Description of Preferred Embodiments

The term "Quality Expert User" refers hereinafter to a user of the device having the privileges to set the standards for acceptable samples of a given checked material.

The term "Operator User" refers hereinafter to a user that uses the device to perform checking of material samples.

The term "Positive sample" refers hereinafter a sample acceptable by the device according to what was set by the Quality Expert User

The term "Negative sample" refers hereinafter a sample not acceptable by the device according to what was set by the Quality Expert User

The term "Master" refers hereinafter the definitions of Positive sample of a certain kind, consisting of reference images and other data items which are stored in the device memory, and used by the device software when the automatic check is performed

As the device is used by manufacturers, traders, and other entities and organizations interested to examine the quality of the discussed materials, effective modus operandi for the device is dividing the users of the single device instance, within an organization, to two types: the Quality Expert Users and the Operator Users.

The Quality Expert Users are persons with adequate knowledge in the quality requirements of the discussed samples. Such users can distinguish between the adequate background knowledge about to configure a corresponding check in the device. As a result, the Quality Expert Users can configure checks in the device that distinguish between Positive and Negative samples of a given material, called Masters.

The Operator Users are persons holding the knowledge of placing the material sample correctly in the device, and triggering the check of the material which is usually triggered by closing the top half of the device and/or pressing one or several buttons in a defined order. Inputting a sample description textually or numerically might be required, for example a batch number, date, time, or any other relevant information which doesn't require the specific knowledge of configuration of checks in the device.

The present invention will be understood from the following detailed description of preferred embodiments, which are meant to be descriptive and not limiting. For the sake of brevity, some well-known features, methods, systems, procedures, components, circuits, and so on, are not described in detail.

The device consists of a portable, fully enclosed sample analysis chamber such as that shown in Fig. la,b. The chamber has two halves 7, 8 which may be opened (Fig. lb) and closed (Fig. la) to allow insertion of a sample. After insertion of a sample the device may be closed as in Fig. la to keep out ambient light. The upper half 7 has a downward-facing hemispherical dome 4, attached by hinges 1 to the lower half 8 which holds the planar analysis surface 2. The dome 4 is provided with lighting means such as RGB, UV, IR, or other LEDs disposed so as to provide uniform illumination of controllable color and intensity to the planar analysis surface when the device is closed.

The device is opened as in Fig. lb and a sample of material (which may be powder, granular, bulk, or the like) is placed on the analysis surface 2 (or on a jig placed on the surface). The device is then closed as in Fig. la and the sample is subsequently analyzed by means of a calibrated camera 3 at the top of the dome. The entire device may be disposed on legs allowing for vibration isolation and height adjustment.

The upper and lower halves are joined when closed in such a manner that no light can enter, for example by using a labyrinthine or multiple-step junction control over various gas pressures, the connection will preferably be made as gas-tight as possible, for example by use of an O-ring and clasps or connectors providing an airtight seal.

It is within provision of the invention that a computing device and associated electronics all be housed within the bottom half 8 of the device. A touchscreen may be attached to top or bottom halves of the device to allow user interaction with the system.

The associated electronics may comprise, inter alia, digital to analog converters for control over the lighting conditions, temperature sensing and control means, weight sensing means, humidity sensing and control means, vibration means adapted to vibrate the sample, pressure sensing and control means, and gas handling apparatus for fixing well-defined partial pressures of a variety of gasses such as oxygen, nitrogen, argon and the like.

It is further within provision of the invention that various chemical agents be released into the sample chamber during or before analysis, either in the gas, liquid, or solid phases. This will allow analysis of reaction between sample and various chemical agents, as for example may be useful in the pharmaceutical industry.

Fig. 2 shows a sectional view of the upper half of the device, including hemispherical dome 4 and lighting sources 5 which may be for instance upward-facing LEDs of variable intensity and/or color.

Fig. 3 shows a further sectional view of the top half and part of the bottom half of the device. A background having defined color and reflectivity 2 lies beneath a sample-holding jig 9 that is bolted 10 in place.

Fig. 4A shows in exploded view the hemispherical cover and positioning stage. The positioning stage has background 2, and jig 9 allowing for precise positioning of samples.

Fig. 4B shows a cutaway view showing the gain correction plate 450, which allows for gain correction due to variations in CCD response, lighting changes, or both.

It is within provision of the invention to use two LED rings for illumination, a first LED ring at the perimeter of the dome, facing upwards and illumination reflect this incoming light evenly down onto the sample.) A second ring may be employed near the top of the dome, these LEDs being downward facing and providing direct illumination of the sample.

The use of two lighting sources in this fashion is of use since one (the upwards facing elements) provide diffuse illumination, while the other (the downwards facing LED ring) provide direct illumination. The hemispherical shape of the top dome allows for the diffuse lighting to illuminate from shallow angles, while the direct illumination allows for prevention of shadows and ample illumination of the background.

Furthermore, the use of both direct and diffuse light sources allows the system to cope with both specular and diffusely reflecting samples.

It is within provision of the invention to vary the intensity of these two types of lighting sources independently, to optimize the lighting on a given sample.

Figs. 4C,D (in side and x-ray top views, respectively) show exemplary embodiments having a ring of upward-facing lighting elements 402, which illuminate the diffusely reflective dome from below. Downward facing lighting elements 401 illuminate the sample directly, from above.

One advantage due to use of the dome is that the sample height may be quite large, for instance a dome having radius of 10cm being capable of admitting samples of 5 cm height without any problem.

It is within provision of the invention to employ a sample positioning jig allowing for precise positioning and holding of various objects such as printed circuit boards, petri dishes, mechanical elements, and the like. The use of such positioning jigs allows for determination of characteristics such accuracy as registration, relative position of elements, and the like.

In one embodiment of the invention a jig system is employed as shown in Fig. 5 A. Here the sample 12 is held in place by arms 13 which may be spring loaded so as to firmly grasp the sample and retain it in a fixed position. The background 2 is visible and provides uniform reflectance at a well-defined color value.

Figs. 5B,C show a further embodiment of the invention adapted for holding samples in position. Jigs 502 (in this case taking the form of rectangular rectangular samples of slightly smaller size fit securely. The sample plate 503 itself fits securely into the device by means of protrusions 501, which define a rectangular area into which the various different sample plates fit securely and repeatably.

It is within provision of the invention that many different jigs may be used with the device, the jigs being replaceable. Thus for example one jig adapted for printed circuit boards of a given size and shape may be employed for a time, and a different jig suited for holding a petri dish or other shape may be used at a different time.

A jig used for purposes of determining the homography or rigid body transformation allowing for precise distance measurement is shown in Figs. 5D, E. Here the protrusions 501 hold the homography plate 511, which has several precisely placed marks 510 which allow the system to determine the rigid body transformation relating the sample plane to the orientation and position of the camera (which is not necessarily exactly perpendicular to the sample plane, and whose distance therefrom is not necessarily known before the homography plate is used to determine these parameters).

For purposes of analysis, conditions of temperature, humidity, lighting conditions, and atmosphere are chosen (by the user or by default). Once the chosen environment has been fixed, analysis may begin. The analysis consists of visual analysis by means of the camera 3, and possibly weight measurement by suitable means in the sample holding tray.

The visual analysis allows for inspection of bulk samples (of grains, powders, etc. ) in terms of shape, color, morphology, and other visual characteristics. The analysis provides for measurement of statistics regarding these characteristics, thereby allowing determination of percentage of a sample that is within allowed tolerances (for size, morphology, color, etc.), sample quality, and other statistical measures that will be clear to one skilled in the art.

As mentioned above, the sample environment may be controlled by means of sensors for and control over lighting, temperature, humidity, pressure, atmosphere, and pressure. Vibration means may be employed to vibrate the sample at a given frequency and amplitude, to allow dispersing of a given connected to the vibration such as stiction, angle of repose, and the like. The vibration means may be supplied within a removable plate such as those providing jigs for positioning samples. Furthermore it is within provision of the invention to separately monitor and control the camera temperature and lighting element temperature, to provide even more exact control over the measurement conditions.

Figs. 6A-C shows an example of one possible provision for vibrating a sample. Fig. 6A is a fully shaded figure while Figs. 6B,C are isometric section views. In these figures one sees the base 603 holding a rubber sheet 602. This sheet in turn holds vibration plate 601 which is in mechanical communication with mechanical actuators 604 such as voice coils, piezoelectric actuators or the like. The vibration plate may itself be outfitted with various sample holding means such as jigs (not shown).

It is within provision of the invention to incorporate sample-holding jigs in conjunction with vibration means. Thus for example a sample table may be supported by vibrating means, or may have vibrating means in mechanical communication therewith, and this table may also be outfitted with jigs or other holding means adapted to (for example) hold a set of petri dishes in place, such that the dishes do not travel during vibration but the contents of the dishes are vibrated all the same.

It is within provision of the invention to use a color standard (or 'gain correction standard') somewhere within the field of view of the camera, to allow for feedback and control over lighting conditions and slight variations between cameras. Thus for example a standardized, diffusely reflecting grayscale bar may be used in each system. This grayscale bar may be used to define for example the middle grey value (e.g. a pixel value of 2 ^A 15=32768 for a 16-bit per pixel camera.) Since the grayscale bar provides an absolute standard, the output from a given camera may be normalized such that it gives the expected value of 32768 even when the raw value may be somewhat different due to camera sensor variations, slight lighting variations and so on. This method may of course also be employed for the various colors of an RGB camera as well. Lighting corrections may also be made in hardware, by (for instance) changing the current supplied to various lighting elements such lighting intensity (for a given color channel) may be varied until the expected value of intensity is detected by the camera. Target plates may be used for the purposes of color correction and also for correcting for spatial aberrations caused by lens distortion of the camera optics. Thus for example a checkerboard of white and black squares of exact dimensions such as 1cm x 1cm for each square may be used as a target plate, and due to the exact and known spacing of squares, images of this plate may be used for an 'image undistortion' calibration which performs an arbitrary nonlinear transformation between the measured pixel coordinates and real- space coordinates.

The geometric calibrations are as follows: A lens undistortion is applied using the [1] calibration method. The geometric transformation from the Image to the Real World Coordinates is performed using Homography estimation. The homography matrix H _3x3 is calculated from N points, fulfilling the equation

^W P = ^W _j H ^{■ 7} P, where ^W P is the point in Homogenous representation in [mm], ⁷ P is the same point on image in Homogenous representation in pixels. After the multiplication is done, the point ( i _; 2, 3)' has to be extracted from homogenous representation by the following scaling operation

The Brightness calibrations are as follows: (1) The dome lighting, top LED lighting, and the lens, create non uniform lighting pattern in the image. To overcome this, as a calibration procedure, a gray diffuse reflective sheet is placed in the working area rig instead of the working background and the image ⁵ / = { ⁵ i¾} is captured, where v are the grayscale values of the pixels at locations (i,j). Then, the Brightness Normalization image ^N I = ^B I/mean({ ^s i¾}) is extracted. In the operation, every image coming from the camera, is corrected as follows: I _corr = I./ ^N I, where ./ means an element wise division.

(2) The changing camera temperature and the LED degradation create a gain effect on the image, which must be compensated for in order to be able to compare brightness information between images of one device taken in different time, or to compare images from different devices. is taken in each captured image. The target plate is situated near the working area. It is attached mechanically to the Top Half of the device, so when the device is open, there is no danger of sullying the target plate. The resulting gain correction works as follows: when defining a reference sample M, the sum of brightness in every channel R, G and B from the pixels belonging to the target plate T is taken, ^M B _c =∑∑ ^M v _c \ _C =R,G,B, { V _C \ V _x , v _y ) G T}.

When capturing an image I of a sample S, the target plate area brightness is calculated in a similar way, B _c =∑∑ ^s v _c \ C=R,G,B - The gain correction coefficient for any channel is found as 5 _C = - _j f^-, S _c G . Finally, the gain corrected image is obtained, l _c ' = l _c l&c-

As will be appreciated by one skilled in the art, the multiple levels of feedback and control over measurement conditions (controlling camera temperature, lighting element temperature, and sample chamber temperature, calibrating for lens distortion, and calibrating for color aberrations) will allow for repeatable and measurements in extremely well -characterized conditions. Thus users of separate instances of the inventive machine will measure precisely the same statistics for a given sample, allowing for direct comparison between (for example) color of textile from a Chinese production line vs. factory output from Bangladesh for the same product.

To deal with bulk samples, the samples may be flatted for instance by knife edge, doctor blade, or a flattening accessory such as that used in preparing espresso shots.

As mentioned above, it is within provision of the device to allow for color analysis of samples. This may take an arbitrarily complex implementation, for example allowing for analysis of patterns in terms of color and spatial variation.

The analysis of color can be used to implement an 'area spectrometer' in the following way. For color control purposes, the color values output by the camera are either measured in or transformed to the 'LAB' palette instead of the more common RGB or CMYK. In this palette, the 'Euclidean distance' between one color value (Li,Ai,Bi) and a second value (L ₂ ,A ₂ ,B ₂ ) may be measured as with the advantage over similar measures in other palettes, that this distance in LAB has physiological import; for instance, it has been found that for colors having a difference D<2, the average human eye cannot distinguish between them. Thus for purposes of quality control in the textile industry, in many cases the rubric D<2 is used as an acceptable maximum variation between product and a master.

In most systems the L,A,B colors are measured at a single point or a small set of points. In contrast the inventive device allows for measurement of such over large areas, allowing for comparison of hundreds or thousands of points. As will be appreciated this allows for significantly better statistics and signal- to-noise ratio than a small set of tens of points.

Image processing means of the invention allow for segmentation of separate objects in an image, and individual measurement of a number of object details such as color, shape, area, perimeter, morphology, curvature, and the like. For example for each visible grain of rice in a sample, the details (color, shape, area, etc) may all be measured and used for statistical analyses, e.g. production of histograms, calculation of averages and variances, etc. Furthermore by means of these details, it is possibly to classify samples or individual sample elements (such as aforementioned individual grains of rice) into various categories such as acceptable vs. not acceptable.

The classification can be accomplished by means of direct input of characteristics (such as minimum/maximum sizes, color ranges, curvatures, etc.) or by means of simply giving examples of 'pass' and 'fail' items. For instance a set of acceptable grains of rice is introduced to the system, analyzed, and stored in software associated with the system as representing acceptable grains. Then a set of unacceptable grains (e.g. discolored, bent, broken, etc) are introduced to the system and analyzed, and then stored in the system as representing unacceptable grains. Based on the statistics of these samples, the definitions of acceptable and unacceptable samples are inferred. This inference may be a simple statistical inference or may employ machine learning techniques such as SVM, neural nets, Bayesian inference, Markov chains, or the like, using one or many samples of any number of categories. a. Particulate samples - a sample that consists of particles, which are a small and separate pieces of material like grains or screws, when the inspection of each particle is important

b. Bulk samples - a sample of a bulk of material. The inspection of the bulk properties is important, but each of the bulk's particles standing alone isn't important for the inspection

c. Powder samples - a sample of a powder, usually having a homogenous color that should be determined more accurately than in a Bulk sample d. Panel samples - a nearly flat panel to be inspected, such as a printed electronic circuit or a keyboard

It is within provision of the invention to use structured light for illumination of the sample, allowing for detailed three-dimensional information to be gathered therefrom. Thus for example a matrix or other collection of points may be projected onto the sample, and their actual positions measured to determine three-dimensional structure. In this regard it may also be useful to vary the position of the sample with respect to the camera. If a known (and preferably small) depth of field is used, the points in focus vs. those not in focus as a function of sample table-camera distance, also allows for production of a three-dimensional map of the sample.

Some exemplary embodiments of a system adapted to provide structured lighting upon the sample plane are shown in Figs. 7A,B,C. Here the camera 702, dome 704, and structured light 705 are shown, the structured light taking the form of a set of lines, grid, points, or other pattern be it regular or irregular. In Fig. 7A the structured light is provided by means of lighting elements 701 which shine through gratings or slits 703, thereby forming pattern 705. Alternatively, Fig. 7B shows an embodiment where a projector 706 is used to directly produce a pattern of structured light 705 of any desired type. As a further example, Fig. 7C shows an embodiment using lasers 710 adapted to produce several beams 711, which together form a structured light pattern 705.

Fig. 8 shows an example of a system of the invention, placed on a table 807 for use by a user 806. The system has top half 801 containing the hemispherical dome 803, bottom half 802 containing target plane 804, a handle 805 adapted to allow user 806 to open the device, and hinges 808 attaching top and bottom halves. consider the diagram of Fig. 9.

First an appropriate background is chosen (appropriate in terms of contrast for example). After placing the background in the lower part of the device, the sample is inserted and the device closed. Then environmental conditions (lighting, temperature, humidity, atmosphere) are set. Brightness normalization and image undistortion are carried out as mentioned above, with use of color, geometry, and other standards in the testing chamber either during analysis or at some other time (e.g. during monthly calibrations of the machine). After normalization and undistortion, background segmentation is done and then the image processing steps of feature calculation and classification. Once classification is finished, results are reported and another sample may be inserted.

As an example of one aspect of results that may be reported by the system consider Fig. 10 wherein a color histogram for one color channel is presented. The percentage exceptional values may be quantified, as can the levels for exceptional values.

Since the system is designed for industrial use, it is produced with an ergonomic design, of high rigidity of design and high ruggedness. This is accomplished for example by use of high-impact plastic and T-slot aluminum profile for the major structural components of the device.

As will be appreciated by one skilled in the art, the image capture of the device may be extremely fast because of the use of an array camera, as opposed to (for example) a linescan camera which may take for example 60 seconds or longer for the formation of a single image.

It will be further appreciated that a wide variety of materials may be used in the device due to the care with which the enclosure, lighting, and vibration means have been designed. For example, large samples of relatively large height (e.g. 10cm) may be accommodated within the dome of the device.

It will be further appreciated that due to the great care with which the color, lighting, and geometry are linearized, normalized, and standardized, that analyses performed by one machine of the invention may be directly compared to results of analyses performed on another machine, in a different location and at a different time, even years apart. This is in contrast to other term changes in output of lighting elements, and input from imaging devices, these long-term changes being inevitable aspects of real systems.

It is within provision of the invention to analyze sample in terms of the information content of various histograms of the sample, such as gray level histograms and the like, taking of the form of histogram entropy and other measures. The foregoing description and illustrations of the embodiments of the invention has been presented for the purposes of illustration. It is not intended to be exhaustive or to limit the invention to the above description in any form. Any term that has been defined above and used in the claims, should be interpreted according to this definition.

The reference numbers in the claims are not a part of the claims, but rather used for facilitating the reading thereof. These reference numbers should not be interpreted as limiting the claims in any form.

References

[1] Heikkila, Janne, and Olli Silven. "A four-step camera calibration procedure with implicit image correction." Computer Vision and Pattern Recognition, 1997. Proceedings., 1997 IEEE Computer Society Conference on. IEEE, 1997.