TECHNICAL FIELD

[0001] The present application generally relates to determining human- object interaction. In particular, but not exclusively, the present application relates to determining human grasp pattern on an object.

BACKGROUND

[0002] This section illustrates useful background information without admission of any technique described herein being representative of the state of the art.

[0003] Automation and robotics, as well as virtual reality applications and simulations are increasingly used in various technical fields. Furthermore, ergonomics of common tasks is an area of interest in view of public health. Accordingly, there is an increasing need to understand how humans interact with objects, as such understanding is applicable in improving for example interaction of robots with similar objects and in designing new objects with improved ergonomics.

[0004] Previously, systems have been presented in which the touch of a human on a surface is detected with thermal imaging in order to provide a touch interface akin to a touch pad or screen on any surface. Such a system has been presented for example in published US patent application US2015/0234454. Furthermore, such systems have been suggested to be used to study human grip patterns e.g. in Heatwave: Thermal Imaging for Surface User Interaction by Eric Larson et. al. in CHI 201 1 , May 7-12, 201 1 in Vancouver, BC, Canada.

[0005] Previous systems are based on thermal imaging with a two dimensional image and of limited use for studying human-object interaction. The object of the present invention is to provide an improved system mitigating the shortcomings of the prior art. SUMMARY

[0006] Various aspects of examples of the invention are set out in the claims.

[0007] According to a first example aspect of the present invention, there is provided a method for determining human-object interaction, comprising

imaging an object to retrieve depth data of the object;

imaging the object to retrieve thermal data;

determining a three dimensional model of the object based on the retrieved depth data;

mapping the retrieved thermal data on the three dimensional model of the object; and

determining grasp data based on the thermal data mapped on the three-dimensional model of the object.

[0008] Imaging the object to retrieve depth data of the object may comprise imaging with at least one depth camera.

[0009] Imaging the object to retrieve thermal data may comprise imaging with at least one thermal camera.

[0010] Imaging the object to retrieve thermal data may comprise imaging with a visible light camera; and the method may further comprise treating the object prior to human-object interaction with thermochromatic material.

[0011] Treating with a thermochromatic material may comprise applying thermochromatic paint, ink, dye or gel, a thermochromatic tape or film, or a thermochromatic coating.

[0012] Determining the three dimensional model may comprise recognizing the object and retrieving a pre-determined model from a database.

[0013] Recognizing the object may comprise recognizing the orientation of the object.

[0014] Determining the three dimensional model may further comprise RGB depth imaging in order to retrieve texture data.

[0015] The grasp data may comprise information on the location and/or strength of a handgrip with which a human has grasped the object.

[0016] Mapping the retrieved thermal data on the three dimensional model of the object may comprise producing a three dimensional thermal map of the object. [0017] The method may further comprise moving the object and the at least one depth camera and the at least one thermal and/or visible light camera relative to each other during imaging.

[0018] According to a second example aspect of the present invention, there is provided a system for determining human-object interaction, comprising

at least one camera configured to imagine an object to retrieve depth data of the object;

at least one camera configured to imagine the object to retrieve thermal data; and

a processor configured to cause

determining a three dimensional model of the object based on the retrieved depth data;

mapping the retrieved thermal data on the three dimensional model of the object; and

determining grasp data based on the thermal data mapped on the three-dimensional model of the object.

[0019] The at least one camera configured to imagine the object to retrieve depth data of the object may comprise at least one depth camera.

[0020] The at least one camera configured to imagine the object to retrieve thermal data may comprise at least one thermal camera.

[0021] The at least one camera configured to imagine the object to retrieve thermal data may comprise at least one visible light camera; and the system may further comprise means for treating the object with a thermochromatic material.

[0022] The thermochromatic material may comprise thermochromatic paint, ink, dye or gel, a thermochromatic tape or film, or a thermochromatic coating.

[0023] Determining the three dimensional model may comprise recognizing the object and retrieving a pre-determined model from a database.

[0024] Recognizing the object may comprise recognizing the orientation of the object.

[0025] Determining the three dimensional model may further comprise RGB depth imaging in order to retrieve texture data.

[0026] The grasp data may comprise information on the location and/or strength of a handgrip with which a human has grasped the object.

[0027] [0028] The system may further comprise means for moving the object and the at least one camera configured to imagine an object to retrieve depth data of the object and the at least one camera configured to imagine the object to retrieve thermal data relative to each other.

[0029] According to a third example aspect of the present invention, there is provided a computer program comprising computer code for causing performing the method of the first example aspect of the present invention, when executed by an apparatus.

[0030] According to a fourth example aspect of the present invention, there is provided a non-transitory memory medium comprising the computer program of the third example aspect of the present invention.

[0031] Different non-binding example aspects and embodiments of the present invention have been illustrated in the foregoing. The embodiments in the foregoing are used merely to explain selected aspects or steps that may be utilized in implementations of the present invention. Some embodiments may be presented only with reference to certain example aspects of the invention. It should be appreciated that corresponding embodiments may apply to other example aspects as well. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] For a more complete understanding of example embodiments of the present invention, reference is now made to the following descriptions taken in connection with the accompanying drawings in which:

[0033] Fig. 1 shows a flow chart of a method according to an embodiment of the invention;

[0034] Fig. 2 shows a flow chart of a method according to an embodiment of the invention

[0035] Fig. 3 a flow chart of a method according to an embodiment of the invention;

[0036] Fig. 4 shows a flow chart of a method according to an embodiment of the invention; and

[0037] Fig. 5 shows a schematic example view of a system according to an embodiment of the invention. DETAILED DESCRIPTON OF THE DRAWINGS

[0038] The present invention and its potential advantages are understood by referring to Figs. 1 through 5 of the drawings. In this document, like reference signs denote like parts or steps.

[0039] Fig. 1 shows a flow chart of a method according to an embodiment of the invention. Human-object interaction takes place at 1 10. In an embodiment, the human-object interaction comprises grasping the object with a hand or hands. In a further embodiment, human-object interaction comprises touching, pressing or pushing the object with hand, fingers or palm. In a still further objection, the human- object interaction comprises interacting with the object with a further body part, such as lips.

[0040] At step 120, subsequent to or during human-object interaction, the object is imaged in order to retrieve depth data of the object and thermal data. In an embodiment, the imaging to retrieve depth data is carried out using at least one depth camera. In a further embodiment, the number of depth cameras is larger than one, for example 2-5 in order to retrieve depth data of the object from all sides. In an embodiment, the depth camera comprises an RGB depth camera using for example Time of Flight or LIDAR principle. In a further embodiment, the RGB depth imaging further comprises retrieving texture data of the object.

[0041] In an embodiment of Fig. 1 , the imaging to retrieve thermal data is carried out using at least one thermal camera. In a further embodiment, the number of thermal cameras is larger than one, for example 2-5 in order to retrieve thermal data of the object from all sides. The human object-interaction changes the temperature of the surface of the object in those parts of the surface which the human has touched, or grasped. The change of temperature, typically the rising of temperature in those parts of the surface of the object which the human has touched albeit relatively small, for example in the range of a few tenths of a kelvin, is detected from thermal data retrieved. The change of temperature of the object surface is for example dependent on the temperature difference between the human and the object and on the duration and force of the touch.

[0042] The depth data retrieved at 120 is used to create a three-dimensional model of the object at 130. In an embodiment, the three dimensional model is created using a suitable computing element, such as a processor of the at least one camera, a processor situated locally in separate element such as a computer or a separate system, for example a cloud based system. In an embodiment, the three- dimensional model, for example a surface model, can be computed from the depth data with surface reconstruction methods such as Poisson surface reconstruction.

[0043] At 140, the thermal data retrieved at 120 is mapped onto the three dimensional model of the object created at 130. In order to form a heat map of the object, i.e. a three-dimensional model comprising thermal data, i.e. the surface temperature. In an embodiment, the mapping of thermal data is carried out using a suitable computing element, such as a processor of the at least one camera, a processor situated locally in separate element such as a computer or a separate system, for example a cloud based system. In an embodiment, the mapping is carried out, for example, by utilizing the knowledge about the camera pose relative to the object at the time of image acquisition.

[0044] At 150, grasp data is determined from the thermal data mapped on the three-dimensional model of the object. In an embodiment, the grasp data comprises information on the positions of the object the human has touched. In a further embodiment, the grasp data comprises information on the duration of the touch. In a still further embodiment, the grasp data comprises information on the applied pressure, i.e. force, of the touch. In an embodiment, the determined grasp data is stored in a grasp data database.

[0045] Fig. 2 shows a flow chart of a method according to an embodiment of the invention. In the embodiment of Fig. 2, the object is at 210 treated with thermochromatic material prior to human-object interaction at 1 10. A thermochromatic material comprises material that changes its color in response to a change of temperature. In an embodiment, the thermochromatic material comprises a thermochromatic paint, ink, dye and/or gel applied to the object surface or to a part of the object surface. In a further embodiment, the thermochromatic material comprises a thermochromatic tape or film applied to the object surface or to a part of the object surface. In a still further embodiment, the thermochromatic material comprises a thermochromatic coating applied to the object surface or to a part of the object surface.

[0046] At step 1 10, the human-object interaction takes place as hereinbefore described with reference to Fig. 1 .

[0047] At step 220, subsequent to or during human-object interaction, the object is imaged in order to retrieve depth data of the object and thermal data. In an embodiment, the imaging to retrieve depth data is carried out using at least one depth camera. In a further embodiment, the number of depth cameras is larger than one, for example 2-5 in order to retrieve depth data of the object from all sides. In an embodiment, the depth camera comprises a RGB depth camera using for example Time of Flight or LIDAR principle. In a further embodiment, the RGB depth imaging further comprises retrieving texture data of the object.

[0048] In an embodiment of Fig. 2, the imaging at 220 to retrieve thermal data is carried out using at least one visible light camera. In a further embodiment, the number of visible light cameras is larger than one, for example 2-5 in order to retrieve thermal data of the object from all sides. The human object-interaction changes the temperature of the surface of the object in those parts of the surface which the human has touched, or grasped, and accordingly, due to the thermochromatic material previously applied to the surface, the color of the surface changes in response to the change of temperature. The change of color is imaged with the visible light camera and the thermal data is retrieved from the change of color.

[0049] In the embodiment of Fig. 2, steps 130 to 150 are carried out as described hereinbefore with reference to Fig. 1 .

[0050] Fig. 3 shows a flow chart of a method according to an embodiment of the invention. In the embodiment of Fig. 3, the three-dimensional object model is formed, or has been formed, beforehand. At 360, the object is provided, and the object is imaged to retrieve depth data of the object at 370. The imaging at 360 to retrieve depth data is carried out as in step 120 hereinbefore described with reference to step Figs. 1 and 2. At 380 a three dimensional model of the object is determined as in step 130 hereinbefore described with reference to Figs. 1 and 2. At step 390, the three dimensional model of the object, i.e. object data, is stored in a database.

[0051] As hereinbefore described, human-object interaction takes place at 1 10 and subsequent to or during human-object interaction at 320, the object is imaged in order to retrieve depth data of the object and thermal data as in step 120 hereinbefore described with reference to Figs. 1 and 2. At 330, the depth data of the object retrieved at 320 is compared to previously determined three dimensional models and accordingly, the object is recognized and a pre-determined three dimensional model of the object is retrieved from a database, i.e. determining the three dimensional model comprises recognizing the object and retrieving a predetermined model from a database. In an embodiment, recognizing the object comprises recognizing the orientation, or pose, of the object from the depth data in order to allow the subsequent mapping of the thermal data. In an embodiment, the database storing pre-determined models is a local database. In a further embodiment, the database storing pre-determined models is located in a separate system, for example a cloud based system.

[0052] Fig. 4 shows a flow chart of a method according to an embodiment of the invention. In the embodiment of Fig. 4, the object is at 210 treated with thermochromatic material prior to human-object interaction at 1 10 as hereinbefore described with reference to Fig. 2. In the embodiment of Fig. 4, the imaging at 420 to retrieve thermal data is carried out using at least one visible light camera. In a further embodiment, the number of visible light cameras is larger than one, for example 2-5 in order to retrieve thermal data of the object from all sides. The human object- interaction changes the temperature of the surface of the object in those parts of the surface which the human has touched, or grasped, and accordingly, due to the thermochromatic material previously applied to the surface, the color of the surface changes in response to the change of temperature. The change of color is imaged with the visible light camera and the thermal data is retrieved from the change of color. The imaging to retrieve depth data of the object is carried out as in step 320 hereinbefore described with reference to Fig. 3.

[0053] Further methods steps in the embodiment of Fig. 4 are carried out as hereinbefore described with reference to Figs. 1 to 3.

[0054] Fig. 5 shows a schematic example view of a system 500 for determining human-object interaction according to an embodiment of the invention. The system 500 comprises at least one depth camera 30a-d and at least one thermal and/or visible light camera 30a-d. In an embodiment, the cameras 30a-d are positioned around the object 10 for example in such a way that three cameras 30a-c are positioned in a horizontal plane around the object and at least one camera 30d is positioned above the object. It is to be noted that the number of cameras 30a-d, the position of the cameras 30a-d and the type of each camera is chosen in accordance with the situation bearing in mind that the system comprises at least one depth camera and at least one thermal and/or visible light camera in accordance with the method to be executed.

[0055] In an embodiment, the object and the at least one depth camera 30a- d and the at least one thermal and/or visible light camera 30a-d are arranged to be movable relative to each other. In an embodiment, the system comprises a pedestal or holder 20 for holding an object 10. In an embodiment, the pedestal is arranged to be rotated, or moved in any dimension, in order to ease the imaging of the object from different sides. In a further embodiment, no pedestal 20 is needed, and the object 10 is imaged in situ in the place in which the object is to be found. It is to be noted that the object 10 in an embodiment comprises a group of objects, for example a control panel or the like with several elements. In a further embodiment, the at least one depth camera 30a-d and the at least one thermal and/or visible light camera 30a-d are arranged to be movable or rotatable around or about the object 10 and/or pedestal 20, for example the at least one depth camera 30a-d and the at least one thermal and/or visible light camera 30a-d is in an embodiment installed in a handheld frame used to scan around the object.

[0056] The system 500 further comprises a control element 40. The control element 40 comprises a processor, processing circuitry or the like configured to cause executing a method according to an embodiment of the invention. The control element 40 further comprises further functionalities and elements assisting in carrying out the methods according to embodiments of the invention, for example but not limited to elements such as a user interface element, communications element, and/or a memory element or elements.

[0057] The control element is connected to the system with wireless or wired connections. In an embodiment, the control element 40 is integrated with the cameras 30a-d or integrated with a separate apparatus, such as a laptop computer. In a further embodiment, the control element, or parts thereof or part of the functionalities thereof, for example memory elements comprising databases, is located in a separate system, such as a cloud-based system.

[0058] Some use cases relating to given embodiments of determining human-object interaction are presented in the following. In a first use case, the method and system according to an embodiment of the invention is used to analyze human-object interaction in order to use the result for training an automated system, such as a robot, to grasp an object in a manner similar to that of a human.

[0059] In a second use case, the method and system according to an embodiment of the invention is used to analyze human-object interaction in an environment, such as a cockpit of an airplane or control room of a power plant, in which a human needs to interact with a multitude of objects. The goal of such an analysis is to provide more ergonomic and easy to use interfaces as well as improved safety and effectiveness.

[0060] In a third use case, the method and system according to an embodiment of the invention is used to analyze human-object interaction in order to determine an optimized design for an object which a human grasps, for example for a tool or a utensil.

[0061] Without in any way limiting the scope, interpretation, or application of the claims appearing below, a technical effect of one or more of the example embodiments disclosed herein is the provision of three dimensional information on human grasping an object. Another technical effect of one or more of the example embodiments disclosed herein is the provision of a flexible and low-cost system for determining human-object interaction. A still further technical effect of one or more of the example embodiments disclosed herein is enabling a precise way of analyzing human-object interaction.

[0062] Although various aspects of the invention are set out in the independent claims, other aspects of the invention comprise other combinations of features from the described embodiments and/or the dependent claims with the features of the independent claims, and not solely the combinations explicitly set out in the claims.

[0063] It is also noted herein that while the foregoing describes example embodiments of the invention, these descriptions should not be viewed in a limiting sense. Rather, there are several variations and modifications which may be made without departing from the scope of the present invention as defined in the appended claims.