TECHNICAL FIELD

An aspect and embodiments thereof relate to a method for recording an activity of a human operator movable object in a laboratory environment. Another aspect and embodiments thereof relate to a method for analyzing activity in a human operated laboratory environment. A further aspect and embodiments thereof relate to a method for visualizing an activity in a human operated laboratory environment.

BACKGROUND

In laboratory environments it is generally required to document every performed act and experimental step. This documentation should be accurate, objective and reliable. Additionally, the documentation should be retrievable for e.g. quality auditing. Often these acts and steps are performed by human operators and the documentation thereof is often done manually.

In order to improve the accuracy within the laboratory environment and to minimize manually documenting data, smart devices have been implemented. These smart devices can measure and display objective data points, such as temperature, pressure, humidity levels, as well as procedural steps that have been taken.

An example is provided in W02020/067900, wherein an add-on device for attachment to a micropipette is described. This add-on device is for determining the rotational movement and longitudinal displacement of a micropipette plunger for digitization, communication and quantification of pipetting actions.

W02006/111977 describes a data acquisition apparatus for use with a fluid dispensing device. The apparatus generally comprises a fluid volume sensor, a data transmission device and a data storage device. WO20 18/141898 describes a handheld fluid transfer apparatus, such as a pipette, comprising a control apparatus, which comprises a data processor, a user interface device, which comprises a motion sensor device, a motion data memory and an evaluation device.

SUMMARY

Disadvantageously, none of the known methods provide a method to record activity data of multiple objects within an at least overlapping time frame such that e.g. the location of the objects with respect to each other can be determined. Accordingly, the known methods do not allow for determining and recording activity data of objects to the extent that this can be used to analyze and/or to visualize the laboratory environment, in particular at a moment of time after the activities have taken place in the laboratory environment, such as minutes, hours, days, or even weeks or longer after the activities have taken place.

To address at least one of the aforementioned drawbacks, a first aspect provides a method for recording an activity of a human operator movable object in a human operated laboratory (herein also referred to as lab) environment.

“Activity” is herein used to describe features that may be attributed to an object. The term “activity” includes features such as a state, a location or a movement of an object.

“The human operator movable object” (herein also referred to as object) is typically a laboratory object. By human operator movable is herein meant that the objects are intended to be movable by a single human operator without excessive force, such as movable by using one or more fingers, one or more hands, and/or one or more arms. Typically, these human operator movable objects are moved at least once during a laboratory procedure. Examples of human operator movable objects, in particular laboratory objects, include fluid containers, such as a beaker glass, a sample vial or an Erlenmeyer, pipette tip holders and handheld objects, such as micropipettes. Micropipette assemblies, wherein the micropipette assembly comprises a micropipette and optionally a tip operatively connected to the micropipette, are also to be comprised by the term human operator movable object.

“State” may be used to describe a characteristic of the object. This state may be expressed as one of two options, such as on or off. It may also be appreciated that a state may be partial, such as a partially filled tip. Examples of states include, but are not limited to:

“depressed” or “not depressed” of a plunger of a micropipette;

“having an empty tip” or “having a filled tip” of a micropipette assembly comprising a tip;

“comprising the tip” or “not comprising the tip” of the micropipette assembly.

“Location” is used to describe the position of an object in space. The location may be expressed by one or more components in three-dimensional space. For instance, the location may be expressed by only a horizontal component, such as a position on a one-dimensional line or on a two- dimensional plane. The location may also be expressed by a vertical component. Any combination of horizontal and/or vertical components may also be used to describe the location. For describing the location of an object, the object may be represented by a single point in space, which point may for example correspond or coincide with a center of the object, or a center of gravity of the object. Alternatively, an object may be represented by multiple points, such as a point cloud or other set of points in two-dimensional or three- dimensional space used to respectively describe a two-dimensional or three- dimensional shape of an object.

“Orientation” is used to describe a direction in which an object is facing, for example in a two-dimensional or three-dimensional space. The orientation of an object may be described as a vector, in particular together with the location as the starting point of the vector. In particular when the object is rotated or pivoted about one or more axes of rotation, the orientation of the object may change.

“Movement” is used to describe a displacement and/or rotation of the object - i.e. respectively a change in location and/or orientation. This movement may be relative to another object, or may be an absolute movement. However, it is also possible that the movement is relative to the laboratory environment. Movement may result in a change in location of the object. The movement may be at least initiated by a force and/or torque applied by the human operator. For instance, the plunger button of a micropipette may be depressed. The reversal movement of the plunger button to its original not depressed state typically does not require human interaction, for example when a biasing element such as a spring is used to bias the plunger button in the not depressed state. However, this movement is considered to be initiated by the human operator as depressing the plunger button is required before the plunger button can move to its not depressed state.

In general, a movement state may be derived using the location and/or orientation state, in particular a change in location state and/or orientation state over time. Additionally or alternatively, a location and/or orientation state may be derived using a movement state. In particular, using a location and/or orientation of an object combined with a movement at a first moment in time, a location and/or orientation of the object at a second moment time after the first moment in time may be obtained. Similarly, a location and/or orientation of an object at two instances in time may be used to derive a movement of said object.

Non-limiting examples of movement include;

- travelled path of a plunger relative to a micropipette;

- travelled path of the ejector button relative to the micropipette;

- travelled path of an object in the laboratory environment;

- velocity or speed of an object at a particular point in time; - rotational velocity or rotational speed of an object at a particular point in time;

- any further derivative of velocity of speed, such as acceleration of an object.

“Human operated lab environment” is used to describe a laboratory environment, wherein at least some, in particular the majority or even all, of the steps of a laboratory protocol require intervention by a human operator. The human operator may use automated processes, such as a centrifuge set to a particular time and revolutions per minute (rpm), however, the human operator places the sample vial in the centrifuge. Therefore such an experimental step is considered requiring intervention of the human operator. The laboratory environment may for example be a single room or space, or even a single lab bench.

“Laboratory protocol” is herein also referred to as lab protocol and used to describe a set of instructions that allows the human operator to conduct an experiment. These experiments may have previously been conducted to ensure optimal instructions or the experiments may not have been conducted before. The instructions may relate to a particular sequence of activities, a particular range or threshold for one or more states, and/or any other constraint on one or more activities and/or one or more states, in any combination thereof.

The method for recording activity data of the object comprises obtaining movement data of the object over a period of time. The period of time may be in the order of milliseconds, seconds, minutes or hours, depending on the experiment or experimental step. The movement data may be indicative of a movement of the object. The movement data may be obtained by a variety of measures, such as cameras and/or movement sensors. The use of cameras may not be desirable due to privacy concerns and regulations. Therefore, it is preferred that the movement data is obtained using movement sensors other than cameras. Movement sensors may include optical sensors, accelerometers, magnetometers and/or gyroscopes. It may be appreciated that the movement data does not have to be directly obtained from the object. For instance, a light beam may be provided which reflects from the surface of the object and is sensed by an optical sensor. The change in reflection may be used as indication of the movement of the object and thus be considered the obtained movement data.

The movement data may be obtained by using a movement sensor directly connected to the object, in particular rigidly connected to the object such that a movement of the movement sensor corresponds to a movement of the object. The sensor may be embedded in the object or generally connected to and/or comprised by the object. It may also be feasible to provide the sensor by an add-on device that can be connected to the object. However, as this may require modification of each individual object, it may be advantageous if the movement sensor is connected or worn by an operator of the object. The operator may have the movement sensor connected or worn by having the sensor provided in e.g. a watch, a glove, an activity tracker, a ring, a sleeve of a jacket, a lab coat, any other type of clothing and/or a bracelet. Such wearables may further be advantageous as data about the size of the object may also be retrieved. For instance, if a glove is used, then the separation between the fingers while holding an object is indicative of the size of the object. When the movement sensor is connector or worn by the operator, the movement of the operator may be used as a proxy for the movement of the object. At least during a movement, a movement of the operator and the object may correspond to one another, for example when the operator rigidly holds the object, and/or applies a force and/or torque on the object.

Once the movement data is obtained, the movement data is stored in a movement data set. At least part of this movement data set is processed to obtain activity data over the period of time. The processing may advantageously be used to reduce or remove any noise in the signal, such that activity data indicative of the activity can be obtained. The activity data may be distinctive for each activity and each object. Subsequently, the activity data is recorded on an electronic memory device.

The electronic memory device used for recording the activity data on is not particularly limiting, any device such as magnetic storage devices, flash memory storage devices, solid-state drives, optical discs, USB flash drives may be suitable. In particular, the electronic memory device comprises a nonvolatile electronic memory. Such non-volatile memory is designed to retain stored information even after power is removed. It will thus be appreciated that the recorded activity data is typically stored such that the activity data can be retrieved at a later point in time. Accordingly, the data should remain recorded and stored, even when power is removed from the electronic memory device. This allows for easy retrieval in case of e.g. quality auditing. The time for which the activity data is recorded on the electronic memory device may be at least 24 hours, at least 7 days or at least 6 months. By having the activity data stored, the method may suitably be used in addition to or as a replacement of a conventional lab journal. In such cases, it may even be preferred to store the activity data for at least 2 years, more preferably at least 3 years.

In general, when activity data is recorded, the movement data processed to obtain said activity data may also be recorded, or the movement data may be discarded after the processing.

Accordingly, the method for recording an activity of a human operator movable object comprises: obtaining movement data of the object over a period of time; storing the movement data in a movement data set; processing at least part of the movement data set to obtain activity data over the period of time; and recording the activity data on an electronic memory device.

By virtue of this method, activity data can be recorded of one or more objects of which typically activity data cannot be recorded directly, for example because there is no electronic control signal available which controls the object as the object is a human operator moveable object which requires a human force and/or torque input to perform its task. Furthermore, the method may allow for one or more activities of the object to be recorded for which no direct sensor is available. Instead, movement data may be used as a proxy for the activity, and processing the movement data allows for obtaining the activity data.

In general, processing at least part of the movement data may comprise one or more steps of filtering movement data. Additionally or alternatively, pattern recognition may be applied to the movement data when processing said movement data. Pattern recognition may for example be based on a trained machine learning algorithm.

In an example of the method, the movement data is indicative of a movement of a plunger of a micropipette, relative to the micropipette comprising the plunger. This movement may be the path from depressed to not depressed or vice versa. The movement may also be a part of the path from depressed to not depressed or vice versa. Such plunger movements typically occur when taking up fluid from a fluid container into the micropipette tip comprised by a micropipette assembly or releasing fluid from the micropipette tip. The movement data indicative of the movement is considered any type of data that provides an indication of how the plunger has moved over the period of time. As detailed above, this data may be obtained by a variety of methods. For instance, a magnetometer may be connected to the plunger button and a magnet to the body of the micropipette. By pressing the plunger button, a change in magnetic field is detected. This detected change in magnetic field can be correlated to the movement of the plunger button relative to the micropipette. As such, the movement data in this case is the magnetic field detected over time. In case the movement sensor comprises an accelerometer, the acceleration of the movement may be detected. This acceleration data may be correlated to the movement of the plunger button, thus the movement data may be considered the accelerometer data in such cases. This movement data is indicative of the movement of the plunger button. As the plunger button is a part of the plunger and travels the same path as the plunger, this movement data is also indicative of the movement of the plunger. At least part of the movement data may be processed to determine the state of the plunger. The state may be “depressed” or “not depressed”. A state of “partially depressed” is nonetheless also possible. This state may then be recorded as activity data, and as such the activity data may be indicative of the state of the plunger over time.

In case the state “partially depressed” is recorded as activity data, it may at a later stage be associated with an error in the performed lab work. Typically, if the plunger is only partially depressed, this indicates that not enough fluid has been taken up or released from the tip of the micropipette assembly. Nonetheless, such a conclusion may typically only be drawn, if it is known that this state was obtained when the micropipette was in close proximity to a fluid container. For this information, activity data of, for instance, a fluid container may also be obtained and compared to the activity data of the plunger (vide infra).

Additionally or alternatively, movement data indicative of a movement of an ejector button relative to a micropipette may be obtained. At least part of this movement data may be processed to determine the state of the ejector button, such as “depressed” or “not depressed”. It may be the case that the ejector button is only “partially depressed”. This may also be considered a state of the ejector button. The state is recorded as activity data. By using at least part of this activity data, in particular the state of the ejector button, one may determine a state of a micropipette assembly such as whether the micropipette comprises a tip or does not comprise a tip. In such cases, the micropipette is comprised by the micropipette assembly. It may be appreciated that determining the state “comprising a tip” or “not comprising a tip” by using the state of the ejector button, may only be reasonable if it is known if the micropipette did or did not have tip before movement data of the ejector button was obtained. It may also be possible to use another set of activity data to determine whether a tip has been operatively connected to the micropipette of a micropipette assembly. For instance by using activity data of a micropipette comprised by a micropipette assembly optionally comprising a tip operatively connected to the micropipette.

In case the micropipette is comprised by a micropipette assembly, the micropipette assembly may further comprise a tip operatively connected to the micropipette. Operatively connected to the micropipette is used to indicate that the tip is attached in a manner that allows for conventional use of the micropipette, i.e. liquid uptake and liquid ejection. In this case, at least part of the activity data of the plunger of the micropipette may be used to determine a state of the micropipette assembly, such as “having an empty tip”, “having a filled tip” or “having a partially filled tip” or even having a particular volume of liquid in the tip, which for example may be expressed in mL. A completely filled tip is typically used to indicate that the amount of fluid in the tip is the amount according to the settings of the micropipette. A partially filled tip may therefore relate to a tip comprising an amount of fluid that is less than what has been set by the volume settings of the micropipette. The activity data of the plunger of the micropipette is in such cases determined by obtaining movement data indicative of movement of the plunger relative to the micropipette comprising the plunger, processing at least part of the movement data to determine the state of the plunger as “depressed” or “not depressed” and by recording this state as activity data. A state of “partially depressed” may also be possible.

The method may alternatively or additionally be used to record activity data of a micropipette comprised by a micropipette assembly. The micropipette assembly may further comprise a tip operatively connected to the micropipette. In this example, the movement data is obtained indicative of a movement of the micropipette. At least part of this movement data is used to determine a state of the micropipette assembly, such as “comprising the tip” or “not comprising the tip”. The determined state may be recorded as activity data. The movement data and optionally the activity data may also comprise information on the acceleration of the micropipette in the lab environment. This acceleration information, particularly the acceleration along a vertical component, may be used to determine whether the tip was attached in a suitable manner. Typically, if the acceleration is too high, the micropipette bumps into the tip with too much force, which may indicate that the tip is not attached in a proper way. The activity data may in such cases be indicative of an error in the performed step.

Additionally or alternatively, movement data may be obtained indicative of a movement of a micropipette. At least part of this movement data may be processed to determine location data indicative of a location of the micropipette in the lab environment. The location data is indicative of the location, which means that the location data may comprise one or more horizontal and/or vertical components of the location within the lab environment. For instance, if the laboratory environment is visualized as a x,y,z coordinate system, then the location data may have a coordinate along one of the axes, such as the x-axis, or along multiple axes. This determined location data indicative of the location of the micropipette is recorded as activity data.

Similarly, movement data indicative of a movement of a fluid container may be obtained. At least part of the movement data is processed to determine the location data indicative of a location of the fluid container in the lab environment. The determined location data indicative of the location of the fluid container may then be recorded as activity data. Herein, the location data may also comprise one or more horizontal and/or vertical components of the location within the lab environment.

The location data indicative of a location of a fluid container may alternatively or additionally be determined by using movement data indicative of a movement of a plunger relative to a micropipette and by using movement data indicative of movement of a micropipette. The movement data indicative of the movement of a plunger relative to a micropipette and the movement data indicative of movement of the micropipette may be obtained by suitable manners as described herein. It is imaginable that if e.g. the plunger has moved to a not depressed state from a depressed state at a particular location of the micropipette, that this location corresponds to a location of a fluid container comprising a fluid that is being taken up. Accordingly, at least part of the movement data indicative of the movement of the plunger relative to the micropipette and at least part of the movement data indicative of the movement of the micropipette is processed to determine location data indicative of a location of a fluid container. Particular examples of fluid containers include a laboratory beaker, a vial or a sample cup, from which fluid may be drawn up and/or into which fluid may be released using the micropipette. The location data indicative of the location of the fluid container may be recorded as activity data.

A similar approach may be taken to determine location data indicative of a location of a pipette tip holder. In this case movement data is obtained indicative of a movement of an ejector button relative to a micropipette and movement data is obtained indicative of a movement of the micropipette. For instance, if the movement of the ejector button corresponds to a state of being depressed at a particular location of the micropipette, one can assume that the micropipette assembly does not comprise a tip. If then at a later point in time, the movement of the micropipette indicates that it has moved downwards with an essentially abrupt end of the path followed by going upwards, then at least the horizontal component of the location of the micropipette is indicative of the location of the pipette tip holder. Accordingly, at least part of the movement data indicative of the movement of the ejector button relative to the micropipette and at least part of the movement data indicative of the movement of the micropipette may be processed to determine location data indicative of a location of a pipette tip holder. This location data indicative of the location of the pipette tip holder may then be recorded as activity data.

The method for recording activity data may also be used to determine whether an object has not been used for a period of time. For instance, if no movement of a micropipette is detected, and no movement data indicative of movement of a micropipette is obtained, this may indicate that the operator is using another object, for example carrying out another step.

Advantageously, by recording activity data of at least one object this activity data can be used to analyze activity in a human operated lab environment. Accordingly, a second aspect is related to a method for analyzing activity in a human operated lab environment, wherein the method comprises recording activity data of a first object using any method for recording activity data according to the first aspect. In other words, the method for analyzing activity in a human operated lab environment at least comprises recording activity data of a first object by: obtaining movement data of the object over a period of time; storing the movement data in a movement data set; processing at least part of the movement data set to obtain activity data over the period of time; and recording the activity data on an electronic memory device.

A comparative data set comprising activity data may further be obtained. This comparative data set may be activity data of a second object obtained using the method for recording activity data as described herein above. Typically, at least one of the first or second object is not a micropipette. Alternatively or additionally, the comparative data set may comprise activity data of a lab protocol. Activity data of a lab protocol may for instance comprise sequences, such as data indicating that a particular step is performed before another step. The activity data of a lab protocol may also comprise ranges, such as a temperature range and/or a time period. Further, the activity data of a lab protocol may comprise constraints or thresholds, such a lower limit for a vertical component of a location. This particular example may be used to indicate that a fluid container has been dropped on the floor.

The method for analyzing the activity further comprises comparing at least part of the activity data of the first object with at least part of the activity data of the comparative data set, in particular over an at least partially overlapping time period.

By being able to compare activity data over an at least partially overlapping time period, it becomes possible to determine whether a lab protocol is followed or has been followed. For example, the analysis to determine whether a lab protocol has been followed is performed randomly to keep an eye on quality. It may also be appreciated that this analysis may also be used when an impurity is found in a product. The comparison may e.g. point out that the temperature was higher than the range prescribed in the lab protocol, that a particular step was skipped, or that one or more steps have been carried out which were not specified in the protocol.

Activity data from a lab protocol may be obtained from an electronic memory device, for example via a local data connection or via an internet data connection.

By comparing activity data of a first object with activity data of a second object over an at least partially overlapping time period, it may also be possible to determine the location of the objects with respect to the other object. This may also be helpful if there is an impurity found in the final product. The activity data may be compared and if the location data indicative of the location of the first object, or at least the horizontal component of the location data, corresponds to the location data indicative of the location of the second object, or at least the horizontal component, one may consider the option that some e.g. some fluid spilled. Accordingly, it is preferred that comparing activity data comprises comparing location data of a location of a first object with location data indicative of a second object, to determine a point in time at which at least part of the location data indicative of the location of the first object corresponds to at least part of the location data indicative of the second object, in particular at least the horizontal components of the location data.

It may be particularly preferred that the method for analyzing activity is used for a micropipette assembly comprising a micropipette, and a fluid container. In this case, the first object is thus a micropipette assembly comprising a micropipette and the second object is a fluid container. Comparing the activity data may comprise comparing location data indicative of a location of the micropipette assembly with location data indicative of a location of the fluid container. This comparison may be used to determine a point in time at which at least part of the location data indicative of the location of the micropipette assembly corresponds to at least part of the location data indicative of the fluid container. In particular, to determine a point in time at which a horizontal component of the location data of the micropipette assembly corresponds to a horizontal component of the location data of the fluid container. In such cases the possibility that fluid from the micropipette assembly may have dripped into the fluid container can be investigated. In some particular cases the vertical component of the location data is also considered. For instance, if a fluid container is located directly above, with identical horizontal components of the location data, a micropipette assembly, then there is generally no chance of spoilage that would affect the final product.

It may be appreciated that the method for analyzing the activity may be performed after one or more experiments have been conducted and/or during said one or more experiments. By analyzing the activity in real-time during the experiment, the human operator may directly be warned if activity is recorded that does not correspond to the lab protocol and/or if activity is recorded that has a chance of affecting the final product, such as a point in time at which at least part of the location data indicative of the location of the micropipette assembly corresponds to at least part of the location data indicative of the fluid container. Such a warning may be any suitable warning, such as a visual warning, e.g. a light turning on, an auditive warning, e.g. a sound, and/or a haptic warning, in any combination thereof. Preferably, analyzing the activity in the lab is performed after the experiment is conducted.

A third aspect relates to a method for visualizing activity in a human operated lab environment. This method comprises retrieving activity data of the first object which has been obtained using any method according to the first aspect from an electronic memory. Said activity data has thus been recorded by at least the following steps: obtaining movement data of the object over a period of time; storing the movement data in a movement data set; processing at least part of the movement data set to obtain activity data over the period of time; and recording the activity data on an electronic memory device.

The recorded activity data may be at least partially retrieved from the electronic memory device and used to visualize the activity of the first object. For instance, if the recorded activity data comprises location data indicative of a location of a micropipette in the lab environment, the travelled path of the micropipette through the lab environment can be visualized. Visualization may take the form of a graph, a simulated picture and/or a simulated movie, preferably shown on a display such as a computer monitor, television display, or virtual/augmented reality headset. This visualization may also be individually stored, such that it can be retrieved at a later point in time.

By virtue of visualizing the activity data, the activity data may be more conveniently evaluated by a human compared to the activity data itself. In particular, the activity data may be in a machine-readable format, for example comprising a large structured quantity of numbers. Similarly, activity data of a second object may be retrieved which has been obtained using any method for recording activity data according to the first aspect as described herein. This activity data may be retrieved from the electronic memory device to visualize the activity of the second object. Advantageously, at least part of the visualization of activity of the first object may be superimposed on the visualization of activity of the second object. Accordingly, a simulation on what has occurred in the lab environment at a particular point in time or over a time period can be provided. The layout of the laboratory environment may accordingly be determined.

It is further possible to use a comparative data set comprising activity data, for instance comprising activity data of a lab protocol, which may be visualized and superimposed on the visualization of the activity of the first and/or second object. This allows for a quick and clear overview of the compliance with the lab protocol.

Visualizing the retrieved activity of the one or more objects may comprise steps of rendering graphical representations of at least one of the one or more objects and/or rendering a graphical representation of at least part of the lab environment. A graphical representation may for example be a photo-realistic representation or a schematic representation.

It will be appreciated that a computer program may be provided that comprises instructions which, when executed by a computer, cause the computer to carry out the method according to the first, second and/or third aspect. This computer program may be stored on a computer-readable data carrier.

Further, one or more of the processing step or steps, comparing step or steps and/or the visualization step or steps of the first, second and/or third aspect may be performed on a processor adapted to perform at least one of those steps. Accordingly, a data processing device may be provided comprising a processor adapted to perform one or more of the following steps in any combination thereof: - processing of at least part of the movement data set to obtain activity data over the period of time according to the first aspect;

- comparing at least part of the activity data of the first object with at least part of the activity data of the comparative data set according to the second aspect;

- visualizing the retrieved activity data of the first object according to the third aspect; and/or

- visualizing the retrieved activity data of the second object superimposed on the visualization of the retrieved activity of the first object according to the third aspect.

In general, visualizing activity data may comprise a step of rendering one or more images, which may be photorealistic or more schematic. Rendering may be performed by an electronic processing device. The rendered one or more images may be shown on an electronic display, for example a computer screen, virtual reality headset, smartphone or tablet screen, or any other electronic display.

Accordingly, the methods provide for a way to record, analyze and/or visualize activity within a lab environment. Preferably, the methods essentially only require sensor data, in particular movement data, and with this data a layout, or a map, of the laboratory environment at a specific point in time, or over a time period, can be obtained. These methods may be suitably used to determine whether a lab protocol is executed correctly. Additionally, these methods provide a way to retrieve if a mistake occurred. For instance, comparing the movement of a micropipette assembly and a fluid container may indicate that the micropipette assembly has passed over the fluid container. Additionally, the methods may be used to determine if this micropipette assembly comprised a tip and if this tip comprised fluid. If this was the case, then passing the micropipette over the fluid container could indicate that a drop of fluid has accidentally fallen into the fluid container, potentially affecting the final product. By having a way to record the activity for a long period of time and a method to analyze and visualize the activity, the method may be used as addition to or as replacement of lab journals that are conventionally updated by hand, for example in writing or using a keyboard for typing. This allows for more time-effective, objective and/or accurate documentation.

BRIEF DESCRIPTION OF THE DRAWINGS

In the figures,

Fig. 1 illustrates a micropipette;

Fig. 2A illustrates a schematic top view of a laboratory environment at a first point in time;

Fig. 2B illustrates a schematic top view of a laboratory environment at a second point in time;

Fig 3A illustrates a first schematic example of movement data of a plunger relative to a micropipette over time;

Fig. 3B illustrates schematic activity data over time obtained from processing the movement data of a plunger relative to a micropipette of Fig. 3A;

Fig 4A illustrates a second schematic example of movement data of a plunger relative to a micropipette over time;

Fig. 4B illustrates schematic activity data over time obtained from processing the movement data of a plunger relative to a micropipette of Fig. 4A;

Fig. 5A illustrates schematic movement data of a micropipette assembly;

Fig. 5B illustrates schematic movement data of a plunger relative to the movement of the micropipette assembly of Fig. 5A; and

Fig. 5C illustrates schematic activity data of the micropipette assembly obtained by processing at least part of the movement data of Fig. 5 A and Fig. 5B. DETAILED DESCRIPTION

A micropipette is known in the art and is used to describe a laboratory instrument that can accurately and precisely transfer volumes of liquid (see e.g. Miller et al. The American Biology Teacher 66(4), 2004, 291- 296). It is thus a fluid handling apparatus. A schematic illustration thereof is presented in Figure 1. In this context it is used to describe a manually operated, handheld micropipette. Micropipettes 1 are available in a variety of sizes and designs, however all comprise a plunger 2, a plunger button 3, a body 4, a tip cone 6, a tip ejector button 7 and a finger hook 8. A volume display 5 may optionally be present. The tip cone 6 can be used to attach a tip, which can be easily ejected using the tip ejector button 7. Together with a tip, the micropipette 1 may form a micropipette assembly (not shown in the figures). The plunger 2 is positioned at least partially in the body 4, such that movement of the plunger 2 relative to the body 4 allows for a pressure difference and subsequent liquid uptake or ejection. Movement of the plunger for liquid uptake or ejection is only along the longitudinal axis of the micropipette i.e. in one dimension. The plunger 2 can be moved by using the plunger button 3. The volume setting may be adjusted e.g. by rotating the plunger, by the plunger button 3 or by another volume adjustment dial (not shown separately) to alter the volume settings, which setting can be read from the optional display 5.

Figure 2A illustrates a schematic top view of a laboratory environment 100 at a first point in time. In the lab environment 100, a lab bench 101 is provided. On the lab bench several human operator movable objects are provided, such as a micropipette 102, fluid containers 103 and a pipette tip holder 104. This schematic top view may be simulated by the method for visualizing an activity in the human operated lab environment according to the third aspect. Activity data of each of the objects may be recorded followed by retrieving at least part of the activity data for each of the objects, in particular by processing movement data of one or more objects. By superimposing the retrieved data of each object, at a particular time or for a particular time frame, a simulated environment can be visualized.

Figure 2B illustrates a schematic top view of a laboratory environment 100 at a second point in time. The laboratory environment of Figure 2A has been changed by changing the location of the micropipette 102. The human operator has been using the micropipette and has placed it back at another spot. Accordingly, this movement may be detected and movement data is obtained over a period of time. The movement data is accordingly stored in a movement data set. The movement data set is processed by a suitable processing method to obtain activity data, and said activity data has been recorded. In this particular case the movement over the period of time is recorded as activity data. This activity data may be used to determine the final location of the micropipette. The path the micropipette has followed over the period of time may be known from the activity data. Thus, if the starting location is known, the final location can also be deduced. Additionally, as the movement data may continuously be obtained over the period of time, the location at each time interval within the period of time can be deduced.

Figures 2A and 2B may be seen as visualizations of activity in a human operated lab environment. In particular when activity data is retrieved of at least one of the objects shown in Figures 2A and 2B indicative of a location of said at least one of the objects, this activity data may be used to graphically render at least part of the visualizations shown in Figures 2A and 2B.

Figure 3A illustrates a first schematic example of movement data of a plunger relative to a micropipette over time. As can be seen from Fig. 3A, the movement data may have some noise. From ti to t2 and from t3 to t4 the movement data of the plunger indicates a significant change, that may be indicative of a substantial movement of the plunger relative to the micropipette. At least part of this movement data may be processed to obtain activity data of the period of time. The obtained activity data is schematically illustrated in Figure 3B. The state of the plunger, being depressed (“D”) or not depressed (“ND”) may be extracted from the movement data and may be stored as activity data.

Figures 3 A and 3B thus illustrate an example wherein noisy movement data is processed to obtain less noisy and preferably discrete activity data here indicative of a state of the plunger. The movement data of the plunger may in any example have been obtained using a movement sensor directly connected to the plunger. Additionally or alternatively, the movement data of the plunger may in any example have been obtained using a movement sensor connected to or worn by an operator of the plunger.

Another schematic example of processing movement data to obtain activity data is provided in Figures 4A and 4B. Herein the movement data of a plunger relative to a micropipette is illustrated in Figure 4A. As can be seen, the movement data between ti and t2 differs from the movement data between t3 and t4. After processing, it may become apparent that the difference in movement data results in a different state of the plunger associated with the respective two time periods. The activity data, obtained by processing the movement data of Figure 4A, is illustrated in Figure 4B. The states of the plunger have been indicated by “ND” (not depressed), “D” (depressed) and “PD” (partially depressed). Thus, indeed from the activity data it may also be determined that the plunger has only been partially depressed in one of the time periods.

A further schematic example of movement data is illustrated in Figures 5A and 5B. In Figure 5A, movement data of a micropipette assembly is illustrated. This movement data may be indicative of the vertical movement of the micropipette assembly. Thus, for instance, the movement data between ti and t2 may indicate that the micropipette assembly has been moved downwards in the direction of the floor. This movement data, or at least a part thereof, may be processed. Figure 5B illustrates movement data of a plunger relative to the micropipette assembly of Figure 5A. As illustrated, the movement of the plunger only has a significant change observed between t3 and t4. This movement data may also, at least in part, be processed. By processing both the movement data indicative of movement of the micropipette assembly and the movement data indicative of the movement of the plunger relative to the micropipette assembly, activity data of the micropipette assembly may be obtained. In this particular example, this activity data is indicative of a state of the micropipette assembly. This state may indicate whether the micropipette assembly comprising a tip or not. This state is recorded as activity data and schematically illustrated in Fig. 5C. In Fig. 5C, the state “NT” corresponds to the micropipette assembly not comprising a tip and the state “T” corresponds to the micropipette assembly comprising a tip.

These states may be determined as the movement data indicative of the movement of the plunger does not provide any significant activity between ti and t2, while the movement data indicative of the movement of the micropipette assembly does provide a significant change within that time period. Thus, from this it may be determined that the micropipette assembly has been moved downwards to pick up a pipette tip from a pipette tip holder. Further, the movement data indicative of the movement of both the micropipette assembly and the plunger illustrates change between t3 and t4. As it is determined from the time period ti to t2 that the micropipette assembly comprises a tip, it may be determined that in the time period between ts and t4 fluid has been taken up by the micropipette assembly, as the micropipette assembly has moved downwards and the plunger has been depressed in that time period.

It may be appreciated that from the same movement data set indicative of the movement of the micropipette assembly, the horizontal components may be processed together with at least part of the movement data indicative of the movement of the plunger relative to the micropipette (Fig. 5B) to determine the location data indicative of a location of a fluid container. As detailed above, it may be determined that in the time period between t3 and t4, a fluid is taken up. Accordingly, the horizontal components of the location of the micropipette at that time period may provide information to determine the location data indicative of a location of a fluid container comprising the fluid that has been taken up by the micropipette assembly. This location data indicative of the location of the fluid container may also be recorded as activity data (not shown).

Additionally, or alternatively, the horizontal components of the movement data indicative of the movement of the micropipette assembly may be processed together with at least part of the movement data indicative of the movement of the plunger relative to the micropipette assembly (Fig. 5B) to determine the location data indicative of a location of a pipette tip holder. As there is no change in movement for the plunger, but there is a vertical movement of the micropipette assembly between ti and t2, it may be determined that within that time period a pipette tip has been operatively connected to the micropipette comprised by the micropipette assembly. The tip is typically taken from a pipette tip holder. Accordingly, the location data indicative of the location of the pipette tip holder may be determined. This location data may be recorded as activity data (not shown).

It is to be noted that the figures are only schematic representations of embodiments that are given by way of non-limiting examples. For the purpose of clarity and a concise description, features are described herein as part of the same or separate embodiments, however, it will be appreciated that the scope of the disclosure may include embodiments having combinations of all or some of the features described.

The word ‘comprising’ does not exclude the presence of other features or steps. Furthermore, the words ‘a’ and ‘an’ shall not be construed as limited to ‘only one’, but instead are used to mean ‘at least one’, and do not exclude a plurality.