The present disclosure refers to a method for operating a laboratory automation system, and a system for operating a laboratory automation system.

Background

Laboratory automation systems are for example employed for determining samples, such as samples of a bodily fluid, in an automated way. Each sample is typically received in a sample container which is processed via the laboratory automation system.

Laboratory automation systems can comprise a plurality of units. A laboratory automation system usually comprises a plurality of laboratory stations or devices such as, for example, a pre-analytical, an analytical or a post-analytical station. Typically, the sample containers are transported on transport carriers between different stations of the system via a sample distribution system or a transportation system. The transport carriers with or without sample containers may be moved along a line for processing the samples. Sample containers may be fed into the laboratory automation system via trays which comprise a plurality of sample containers. A placement device may pick a sample container from the tray and place the sample container in a reception place of a transport carrier or vice versa.

Document US 2018 I 0 045 747 A1 discloses a method of tube slot localization using a tray coordinate system and a camera coordinate system. The method includes receiving, a series of images from at least one camera of a tray comprising tube slots arranged in a matrix of rows and columns. The method includes determining calibration information to provide mapping of locations from the tray coordinate system to locations from the camera coordinate system and automatically aligning the tray based on the encoder value and calibration information.

Document US 10 319 092 B2 relates to a method for detecting properties of sample tubes that includes extracting image patches substantially centered on a tube slot of a tray or a tube top in a slot. For each image patch, the method includes assigning a first location group defining whether the image patch is an image center, a corner of an image or a middle edge of an image, selecting a trained classifier based on the first location group, and determining whether each tube slot contains a tube. Document US 10 290 090 B2 discloses a method of using image-based tube top circle detection that includes extracting, from one of a series of images of a tube tray, a region of interest patch having a target tube top circle and boundaries constrained by two-dimensional projections of different types of tube top circle centers.

Document US 10 140 705 B2 relates to methods and systems for detecting properties of sample tubes in a laboratory environment including a drawer vision system that can be trained and calibrated.

In document WO 2019 013 960 A1, methods for image-based detection of the tops of sample tubes used in an automated diagnostic analysis system are disclosed that are based on a convolutional neural network to pre-process images of the sample tube tops to intensify the tube top circle edges while suppressing the edge response from other objects that may appear in the image.

Document EP 3 581 935 A1 pertains to a method for operating a system for the automatic detection of laboratory work items and their state in the area of target storage positions of a laboratory work area.

Document US 10 274 505 B2 is related to an analytical laboratory system and a method for processing samples, in which a sample container is transported from an input area to a distribution area by a gripper comprising a means for inspecting a tube. An image is captured of the sample container and analyzed to determine a sample container identification.

In document WO 2019 084 468 A1 , systems and methods are provided for recognizing various sample containers carried in a rack. Sample containers in the rack are identified and various characteristics associated with the containers and the rack are detected and evaluated to determine the validity and types of the containers in the rack.

Document US 9 135 515 B2 discloses a method for visually inspecting pelletized samples. The method generates an inspection image of the bottom of a sample tube holder.

The system in document EP 2 776 844 B1 includes an image acquisition device configured to obtain one or more images of sample containers in a sample container holder and an image analysis device coupled to the image acquisition device. In document US 10 509 047 B2, a method and a device for handling sample tubes are presented. A position of the sample tube is identified and the sample tube is handled depending thereon.

Document EP 3 330 713 A1 relates to a laboratory handling system, comprising a number of racks, each adapted to carry a number of laboratory sample containers, a rack placement unit, wherein a rack is placed on the rack placement unit, and a rack sensing unit, wherein the rack sensing unit is adapted to sense a position and / or a type of the rack being placed on the rack placement unit.

Document US 2017 124 704 A1 discloses methods and systems for detecting properties of sample tubes in a laboratory environment including a drawer vision system that can be trained and calibrated. Images of a tube tray captured by at least one camera are analyzed to extract image patches that allow a processor to automatically determine if a tube slot is occupied, if the tube has a cap, and if the tube has a tube top cup. The processor can be trained using a random forest technique and a plurality of training image patches. Cameras can be calibrated using a three-dimensional calibration target that can be inserted into the drawer.

Summary

It is an object of the present disclosure to provide a method for operating a laboratory automation system and a laboratory automation system with improved flexibility and safety regarding the placement of sample containers.

For solving the problem, a method for operating a laboratory automation system according to independent claim 1 is provided. Further, a system for operating a laboratory automation system according to independent claim 16 is provided. Further embodiments are disclosed in dependent claims.

According to one aspect, a method for operating a laboratory automation system is provided, the laboratory automation system comprising a carrier comprising a reception place for receiving a sample container configured to contain a sample to be analyzed by a laboratory device; a placement device configured to pick and place the sample container; an imaging device; and a data processing device comprising at least one processor and a memory. The method comprises detecting an image of the reception place by the imaging device; deter- mining whether the reception place is free for receiving the sample container, and whether the reception place is configured to receive the sample container, by applying a machine learning algorithm for image analysis of the image of the reception place in the data processing device; and placing the sample container in the reception place by the placement device if the reception place is determined as free and configured to receive the sample container.

According to another aspect, a laboratory automation system is provided. The laboratory automation system comprises a carrier comprising a reception place for receiving a sample container configured to contain a sample to be analyzed by a laboratory device; a placement device configured to pick and place the sample container; an imaging device; and a data processing device comprising at least one processor and a memory; and is configured to detect an image of the reception place by the imaging device; determine whether the reception place is free for receiving the sample container, and whether the reception place is configured to receive the sample container, by applying a machine learning algorithm for image analysis of the image of the reception place in the data processing device; and place the sample container in the reception place by the placement device if the reception place is determined as free and configured to receive the sample container.

As result, errors in the laboratory automation system due to placing a wrong type of sample container or carrier into a machine, placing at wrong position, inaccurate placing, or not leaving a reception place free can be handled and I or prevented.

Within the context of the present disclosure, a carrier may generally refer to a device for receiving one sample container or a plurality of sample containers. The carrier may also refer to a rack or a tray.

The system may comprise the laboratory device. The laboratory device may e.g. be configured for analyzing samples. The sample may contain a bodily fluid.

The reception place may comprise a recess (slot) within the carrier. Additionally or alternatively, the reception place may comprise a protrusion on the carrier. The reception place may have a reception place bottom surface and a reception place lateral surface. A reception place depth of the reception place may be a vertical difference between the reception place bottom surface and a top edge of the reception place. A reception place bottom surface shape and / or a reception place top surface shape may be one of a circular disc, a polygon, a rectangle, a square, and an ellipse. A reception place bottom angle between the reception place bottom surface and the reception place lateral surface may be greater than or equal to 90°, preferably greater than or equal to 90° and less than 150°, more preferably greater than or equal to 90° and less than 100°, most preferably greater than or equal to 90° and less than 95°.

The reception place, in particular the slot, may have a one of a cylindrical shape, a conical shape, a polyhedral shape, a cubic shape, and a cuboid shape. The cylindrical shape may be one of a circular cylindrical shape, an elliptical cylindrical shape, and a prism shape.

The sample container may have a container bottom surface, a container lateral surface, and a container top surface. A container height of the sample container may correspond to the vertical difference between the container bottom surface and the container top surface.

A container bottom surface shape and / or a container top surface shape may be one of a circular disc, a polygon, a rectangle, a square, and an ellipse. A container bottom angle between the container bottom surface and the container lateral surface may be greater than or equal to 90°, preferably greater than or equal to 90° and less than 150°, more preferably greater than or equal to 90° and less than 100°, most preferably greater than or equal to 90° and less than 95°. An angle difference between the reception place bottom angle and the container bottom angle may be less than 30°, preferably less than 10°. The reception place bottom angle and the container bottom angle may be the same.

The sample container may have a one of a cylindrical shape, a conical shape, a polyhedral shape, a cubic shape, and a cuboid shape. The reception place and the sample container may have the same shape or a different shape. The sample container may be a sample tube.

The determining whether the reception place is configured to receive the sample container may comprise determining a reception place type from the image using the machine learning algorithm.

The determining whether the reception place is configured to receive the sample container may be carried out in reaction to determining whether the reception place is free for receiving the sample container. In particular, the determining whether the reception place is configured to receive the sample container may be carried out in reaction to having determined that the reception place is free for receiving the sample container. In other words, it may be provided that determining whether the reception place is configured to receive the sample container is carried out (only) for reception places free for receiving the sample container.

It may also be provided that determining whether the reception place is free for receiving the sample container and determining whether the reception place is configured to receive the sample container is carried out simultaneously, in particular by applying the machine learning algorithm for image analysis of the image of the reception place in the data processing device,

The determining of the reception place type from the image may comprise classifying the reception place type by the machine learning algorithm. In this or other embodiments, the machine learning algorithm may for example comprise an artificial neural network. The machine learning algorithm may comprise a supervised learning algorithm, such as a support vector machine, decision tree learning, or similarity learning. The machine learning algorithm may also comprise a pattern matching algorithm. The reception place type may for example be associated with a class label.

The method may further comprise comparing the reception place type with a container type of the sample container, and determining the reception place being configured to receive the sample container if the reception place type is assigned to the container type. In an embodiment it may be provided that each reception place type is assigned to at least one, preferably exactly one container type. The reception place type may be assigned to the container type if the reception place type and the container type match geometrically.

In another embodiment, the method may further comprise the reception place type indicating a reception place bottom diameter being less than a bottom diameter upper bound and I or greater than a bottom diameter lower bound. The reception place type may also comprise a reception place top diameter being less than a top diameter upper bound and I or greater than a top diameter lower bound. The reception place bottom diameter may be the diameter of the reception place bottom surface. The reception place top diameter may be the diameter of the area surrounded by a top edge of the reception place.

The bottom diameter upper bound may be between 7 mm and 20 mm, preferably between 9 mm and 15 mm, more preferably 13 mm (depending on the reception place type). The bottom diameter lower bound may be between 6 mm and 19 mm, preferably between 8.5 mm and 14.5 mm, more preferably 12.8 mm. The top diameter upper bound may be between 7 mm and 20 mm, preferably between 9 mm and 15 mm, more preferably 13 mm. The top diameter lower bound may be between 6 mm and 19 mm, preferably between 8.5 mm and 14.5 mm, more preferably 12.8 mm.

The bottom diameter upper bound may be less than the top diameter upper bound. Alternatively, the bottom diameter upper bound and the top diameter upper bound may be same. The bottom diameter lower bound may be less than the top diameter lower bound. Alternatively, the bottom diameter lower bound and the top diameter lower bound may be same. The bottom diameter lower bound may be less than the bottom diameter upper bound. Alternatively, the bottom diameter lower bound and the bottom diameter upper bound may be the same. The top diameter lower bound may be less than the top diameter upper bound. Alternatively, the top diameter lower bound and the top diameter upper bound may be the same.

The reception place may also be determined as being configured to receive the sample container if at least a first difference of the reception place bottom diameter and the container bottom diameter is between 0.0 % and 5.0 % of the reception place bottom diameter, preferably between 0.5 % and 2.0 % of the reception place bottom diameter, more preferably 1.0 % of the reception place bottom diameter. The first difference may alternatively be between 0 % and 5.0 % of the container bottom diameter, preferably between 0 % and 2.0 % of the container bottom diameter, more preferably 0.1 % of the container bottom diameter. Further alternatively, the first difference may be between 0 mm and 4.0 mm, preferably between 0 mm and 2.0 mm, more preferably 0.1 mm.

The method may further comprise the reception place type indicating a reception place depth being less than a depth upper bound and I or greater than a depth lower bound. The depth upper bound may be between 50 mm and 210 mm, preferably between 80 mm and 180 mm, more preferably 100 mm. The depth lower bound may be between 40 mm and 200 mm, preferably between 75 mm and 175 mm, more preferably 99 mm. The depth lower bound may be less than the depth upper bound. Alternatively, the depth lower bound and the depth upper bound may be the same.

The reception place may also be determined as being configured to receive the sample container if at least a second difference between the reception place depth and the container height is between 1 % and 80 % of the reception place depth, preferably between 5 % and 50 % of the reception place depth, more preferably 20 % of the reception place depth. The second difference may also be between 1 % and 80 % of the container height, preferably between 5 % and 50 % of the container height, more preferably 20 % of the container height. Further alternatively, the second difference may be between 1.0 mm and 5.0 cm, preferably between 5.0 mm and 2.5 cm, more preferably 2.0 cm.

In an embodiment, the method may further comprise the reception place type indicating a reception place bottom angle, between a reception place bottom surface and a reception place lateral surface of the reception place, being less than a bottom angle upper bound and I or greater than a bot-tom angle lower bound. The bottom angle upper bound may be between 90° and 150°, preferably between 90° and 110°, more preferably between 90° and 95°. The bottom angle lower bound may be between 89° and 170°, preferably between 90° and 105°, more preferably between 90° and 92°. The reception place type may comprise a reception place top angle being less than a top angle upper bound and I or greater than a top angle lower bound.

The top angle upper bound may be between 90° and 150°, preferably between 90° and 110°, more preferably between 90° and 95°. The top angle lower bound may be between 89° and 170°, preferably between 90° and 105°, more preferably between 90° and 92°.

The reception place type may further comprise at least one of a reception place bottom surface shape, a reception place top surface shape, a reception place lateral surface shape, and a reception place notch indicator indicative of a presence of a lateral notch at the reception place.

The method may also comprise determining at least one of the reception place bottom surface shape, the reception place top surface shape, the reception place bottom diameter, the reception place top diameter, the reception place depth, the reception place bottom angle, the reception place top angle, the reception place notch indicator from the determined reception place type. Further, the method may comprise comparing the at least one of the reception place bottom surface shape, the reception place top surface shape, the reception place bottom diameter, the reception place top diameter, the reception place depth, the reception place bottom angle, the reception place top angle, the reception place notch indicator from the determined reception place type with a corresponding container type value. From the comparing, the reception place may also be determined as being configured to receive the sample container. In general, at least one of the reception place bottom surface shape, the reception place top surface shape, the reception place bottom diameter, the reception place top diameter, the reception place depth, the reception place bottom angle, the reception place top angle, the reception place notch indicator, the lateral positioning, and the reception place being free for receiving the sample container may be determined by applying the machine learning algorithm.

The reception place being free for receiving the sample container may in particular comprise the reception place being empty, in particular, the reception place not having received a (further) sample container. The reception place not being free for receiving the sample container may comprise the reception place not being empty, in particular, the reception place having received a (further) sample container.

The method may comprise providing container type data indicative of the container type in the memory. The container type data may be indicative of at least one of the following: container bottom surface shape, container top surface shape, container bottom diameter, container top diameter, container height, container bottom angle, container top angle, and a container notch indicator indicative of a presence of a lateral notch at the container.

The method may comprise: scanning the sample container by a scanning device; and, in response to the scanning of the sample container by the scanning device, providing the container type from a database. The scanning may comprise scanning a tag of the sample container. The tag may for example be a bar code or a QR code.

The database may be stored in the memory of the data processing device. Alternatively, the database may be stored in an external data processing device and, preferably, be transmitted from the external data processing device to the data processing device. The database may comprise assignments of reception place types to container types and I or assignments of container types to reception place types.

Alternatively, the container type may be provided via time scheduling information and I or location information data. For example, sample containers of a specific container type (corresponding to a specific reception place type) may be provided at a specific time and / or at a specific location. The container type may be provided from the database. The method may further comprise: detecting a second image of the reception place by the imaging device; and determining the container type from the second image using the machine learning algorithm. Alternatively, the method may further comprise detecting the second image of the reception place by a second imaging device different from the imaging device, wherein the providing of the container type may comprise determining the container type from the second image using the machine learning algorithm. The second imaging device may be a camera. The image and I or the second image may be transmitted from the imaging device (respectively, the second imaging device) to the data processing device.

Placing the sample container in the reception place may comprise moving the sample container by the placement device in a horizontal and / or a vertical direction. The placement device may comprise a gripping device for mechanically gripping and I or releasing the sample container.

If the reception place is determined being not free and I or not configured (not suitable) to receive the sample container, an error signal may be generated by the data processing device. In this case, the sample container may not be placed in the reception place.

The method may further comprise determining, from the image, a lateral positioning of the reception place, e.g., via the machine learning algorithm or a further machine learning algorithm. The lateral positioning may for example be with respect to at least one of the carrier, the placement device, a distribution device containing the placement device, and the laboratory automation system. Determining the lateral positioning of the reception place may comprise determining a center point of the reception place. The reception place may be determined as not configured to receive the sample container if the center point of the reception place is offset from a reference point beyond a threshold value.

The detecting of the image may comprise detecting the image of a single reception place. Alternatively, detecting the image may comprise detecting the image of a plurality of reception places. The image may comprise a top view of the reception place. The image may also comprise a side view, a bottom view, or a diagonal view of the reception place.

The method may comprise providing the imaging device attached to the placement device. In particular, the imaging device may be configured to move together with the placement device during operation of the placement device. Alternatively, the imaging device may be (physically) separate from the placement device. The imaging device may for example be a camera. The method may further comprise determining at least one trained pattern via the machine learning algorithm using training images indicative of reception places, in particular indicative of reception place types. The training may be carried out in the data processing device. Alternatively, the training may be carried out in a further external data processing device. A trained model may then be transmitted from the further external data processing device to the data processing device.

The determining of the at least one trained pattern may comprise processing training images indicative of different reception place types, in particular with different reception place bottom surface shapes, reception place top surface shapes, reception place bottom diameters, reception place top diameters, reception place depths, reception place bottom angles, reception place top angles, and reception place notch indicators. Further, the training images may be indicative of the reception place either being free or not being free for receiving the sample container.

The determined trained pattern(s) I trained model(s) may be provided in the data processing device. Determining whether the reception place is configured to receive the sample container may comprise processing the at least one trained pattern. In particular, classifying the reception place type may comprise processing the at least one trained pattern.

The placing of the sample container in the reception place may comprise verifying an expected container height of the sample container by a (e.g. vertical) movement of the placement device. Preferably, a further trained pattern may be determined via the machine learning algorithm using a verification result of the expected container height. Alternatively, the at least one trained pattern is further determined via the machine learning algorithm using the verification result of the expected container height.

Verifying the expected container height of the sample container may for example comprise recording a vertical distance by which the placement device has been moved for placing the sample container into the reception place. From the vertical distance, a measured container height of the sample container can be determined, e.g., by subtracting an offset value from the vertical distance. The offset value may be dependent on at least one of a first default vertical positioning of the placement device, a second default vertical positioning of the carrier, a carrier height, and a vertical gripping position of the placement device when gripping the sample container for placement, in particular a vertical gripping position of the placement device relative to the sample container.

The expected container height may be determined from the scanning of the sample container by the scanning device and I or by determining the container type from the second image.

It may be provided that in case the measured container height is equal to or within a permissible limit of the expected container height, the verification result is assigned as positive. Otherwise, the verification result may be assigned as negative. The permissible limit may for example be less than 5 % of the expected container height, preferably less than 1 % of the expected container height.

The embodiments described above in connection with the method for operating a laboratory automation system may be provided correspondingly for the laboratory automation system.

Description of further embodiments

In the following, further embodiments, by way of example, are described with reference to figures. In the figures show:

Fig. 1 a schematic representation of a laboratory automation system;

Fig. 2 a representation of a carrier and a sample container from a side view; and Fig. 3 a representation for method for operating a laboratory automation system.

Fig. 1 shows a graphical representation of a laboratory automation system 1. The laboratory automation system 1 comprises a carrier 10 comprising a reception place 11 for receiving a sample container 12 configured to contain a sample to be analyzed by a laboratory device 13. The laboratory automation system further comprises a placement device 14 configured to pick and place the sample container 12, an imaging device 15, such as a camera attached to the placement device 14, and a data processing device 16 comprising at least one processor 17 and a memory 18. Further, a scanning device 19 (for example, a bar code scanner) can be provided.

The placement device 14 comprises a gripping device for mechanically gripping the sample container 12 in order to pick and place the sample container 12. Fig. 2 shows a graphical representation of the carrier 10 (with the reception place 11) and the sample container 12 from a side view. In the shown embodiment, the reception place 11 corresponds to a recess (slot) within the carrier 10. The reception place 11 has a reception place bottom surface 20, a reception place lateral surface 21 , and a reception place top edge 22. A reception place depth of the reception place 11 corresponds to the vertical difference between the reception place bottom surface 20 and the reception place top edge 22.

In case a reception place bottom angle, the angle between the reception place bottom surface 20 and the reception place lateral surface 21 , is 90°, the reception place depth is equal to the vertical length of the reception place lateral surface 21. The reception place bottom surface 20 may be a circular disk. If in this case the reception place bottom angle is 90°, the reception place 11 has a (circular) cylindrical shape. A reception place bottom width, a reception place bottom length, and a reception place bottom diameter may respectively correspond to the width, length, and diameter of the reception place bottom surface 20. A reception place top width, a reception place top length, and a reception place top diameter may respectively correspond to the width, length, and diameter of the area surrounded by the top edge 22.

A reception place top angle corresponds to the angle between reception place lateral surface 21 and a carrier top surface 26.

Correspondingly, the sample container 12 has a container bottom surface 23, a container lateral surface 24, and a container top surface 25. A container height of the sample container 12 corresponds to the vertical difference between the container bottom surface 23 and the container top surface 25. The container bottom surface 23 may be a circular disk. In this case, the sample container 12 may have a (circular) cylindrical shape.

A container bottom width, a container bottom length, and a container bottom diameter respectively correspond to the width, length, and diameter of the container bottom surface 23. A container top width, a container top length, and a container top diameter respectively correspond to the width, length, and diameter of the container top surface 25.

Fig. 3 shows a graphical representation of a method for operating a laboratory automation system. In a first step 31, trained patterns are determined using a machine learning algorithm, which may, e.g., comprise an artificial neural network. Determining the trained patterns may comprise processing training images with different reception place types, e.g. comprising different reception place bottom surface shapes, reception place top surface shapes, reception place bottom diameters, reception place top diameters, reception place depths, reception place bottom angles, reception place top angles, and reception place notch indicators. The trained patterns are provided in the first data processing device 16.

In a second step 32, the image of the reception place 11 is detected by the imaging device 15. To this end, the imaging device 15 attached to the placement device 14 may be moved into a position above the carrier 10 with the reception place 11.

In a third step 33, the data processing device 16 determines whether the reception place 11 is free for receiving the sample container 12 and whether the place type of the reception place 11 is configured to receive the sample container 12. Hereto, the image of the reception place 11 is analyzed using a machine learning algorithm image analysis for determining the determining the reception place type.

In a fourth step 34, in reaction to the reception place 11 being determined as both free and configured to receive the sample container 12, the sample container is placed in the reception place 11 by the placement device 14.