CROSS-REFERENCE TO RELATED APPLICATION

[ 0001] The present disclosure claims priority to Singapore patent application No. 10202106806S filed with IPOS on 22 June 2021, the content of which is incorporated herein by reference in its entirety.

TECHNICAL FIELD

[ 0002] Embodiments of the disclosure relate to the technical field of image processing, and particularly, but not limited, to a body and hand association method and apparatus, a device, and a storage medium.

BACKGROUND

[ 0003] In a complex scene with many people, there is usually such a phenomenon that a person is occluded by another person or arms are crossed. In the related art, a person that a hand belongs to is usually judged by use of a sensor on the basis of detecting a hand key point using Human Keypoints Detection (HKD), which cannot ensure the judgment accuracy, and is relatively high in judgment cost.

SUMMARY

[ 0004] The embodiments of the disclosure provide technical solutions to body and hand association.

[ 0005] The technical solutions of the embodiments of the disclosure are implemented as follows.

[ 0006] The embodiments of the disclosure provide a body and hand association method, which may include the following operations.

[ 0007] An image to be detected of which an image content includes a body and a hand is acquired.

[ 0008] A body bounding box of the body and a hand bounding box of the hand are determined in the image to be detected.

[ 0009] An association probability between the body bounding box and the hand bounding box is determined.

[ 0010] A circumscribed box of the body bounding box and the hand bounding box is determined.

[ 0011] A degree of association between the body and hand in the circumscribed box is determined based on a hand key point in the circumscribed box and the association probability. In this manner, the degree of association between the body and hand in the circumscribed box is determined based on the association probability between the body bounding box and the hand bounding box and the hand key point in the circumscribed box of the body bounding box and the hand bounding box, so that the accuracy in association of the body bounding box and the hand bounding box in the circumscribed box may be improved.

[ 0012] In some embodiments, the operation that the association probability between the body bounding box and the hand bounding box is determined may include that: a body feature of the body bounding box and a hand feature of the hand bounding box are determined; and the association probability between the body bounding box and the hand bounding box is determined based on the body feature and the hand feature. In this manner, the association probability between the body bounding box and the hand bounding box is determined based on the body feature and the hand feature, so that the accuracy of determining an association between the body bounding box and the hand bounding box may be improved.

[ 0013] In some embodiments, the operation that the circumscribed box of the body bounding box and the hand bounding box is determined may include that: a body bounding box and a hand bounding box between which an association probability is greater than a preset probability threshold are determined as a target bounding box pair; and a circumscribed box of the body bounding box and the hand bounding box in the target bounding box pair is determined. In this manner, the circumscribed box of the corresponding body bounding box and hand bounding box is determined based on the association probability, so that a circumscribed box of a body bounding box and a hand bounding box between which a degree of association is relatively high may be determined efficiently and accurately.

[ 0014] In some embodiments, the operation that the circumscribed box of the body bounding box and the hand bounding box in the target bounding box pair is determined may include that: a candidate box that encloses the target bounding box pair is determined; and a cover area of the candidate box is adjusted based on a relative positional relationship between the body bounding box and the hand bounding box in the target bounding box pair to obtain the circumscribed box. In this manner, the circumscribed box of the body bounding box and the hand bounding box is determined based on the candidate box that may enclose the body bounding box and the hand bounding box, so that a more accurate circumscribed box may be determined rapidly and efficiently, and meanwhile, the accuracy of determining the association between the body bounding box and the hand bounding box that form the circumscribed box may be improved based on the hand key point in the more accurate circumscribed box.

[ 0015] In some embodiments, the operation that the degree of association between the body and hand in the circumscribed box is determined based on the hand key point in the circumscribed box and the association probability may include that: at least two target hand key points in the circumscribed box are determined, the at least two target hand key points including key points that belong to two different hands of a same person respectively; and the degree of association between the body bounding box and the hand bounding box that form the circumscribed box is determined based on the at least two target hand key points. In this manner, hand key points in the circumscribed box are comprehensively detected, so that the accuracy in association of the body bounding box and the hand bounding box that form the circumscribed box may further be improved.

[ 0016] In some embodiments, the operation that the degree of association between the body bounding box and the hand bounding box that form the circumscribed box is determined based on the at least two target hand key points may include that: a confidence that each target hand key point is a ground truth point is determined; in a case that any confidence is greater than a preset threshold, a distance from the target hand key point corresponding to the confidence greater than the preset threshold to the hand bounding box that forms the circumscribed box is determined; a target association probability between the body bounding box and the hand bounding box that form the circumscribed box is determined based on the distance; and the degree of association between the body bounding box and the hand bounding box that form the circumscribed box is determined based on the target association probability. In this manner, if the confidence that the target hand key point is a ground truth point is greater than the preset threshold, the accuracy in association of the body bounding box and the hand bounding box that form the circumscribed box may further be improved based on a more accurate parameter through the distance from the target hand key point to the hand bounding box that forms the circumscribed box.

[ 0017] In some embodiments, the operation that the target association probability between the body bounding box and the hand bounding box that form the circumscribed box is determined based on the distance may include that: in a case that the distance is less than or equal to a preset distance, the association probability between the body bounding box and the hand bounding box that form the circumscribed box is determined to obtain an in-box association probability; and the in-box association probability is determined as the target association probability. In this manner, a numerical value of the distance may be judged to determine the target association probability between the body bounding box and the hand bounding box rapidly and conveniently, and meanwhile, unnecessary calculations may be reduced.

[ 0018] In some embodiments, the operation that the target association probability between the body bounding box and the hand bounding box that form the circumscribed box is determined based on the distance may include that: in a case that the distance is greater than the preset distance, the association probability between the body bounding box and the hand bounding box that form the circumscribed box is decreased to obtain a corrected association probability; and the corrected association probability is determined as the target association probability. In this manner, the numerical value of the distance may be judged to adjust the related parameter rapidly and conveniently to determine the target association probability between the body bounding box and the hand bounding box, and meanwhile, unnecessary calculations may be reduced.

[ 0019] In some embodiments, the operation that the association probability between the body bounding box and the hand bounding box that form the circumscribed box is decreased to obtain the corrected association probability in the case that the distance is greater than the preset distance may include that: in the case that the distance is greater than the preset distance, a preset adjustment parameter configured to adjust the association probability is acquired; and the association probability between the body bounding box and the hand bounding box that form the circumscribed box is lowered using the preset adjustment parameter to obtain the corrected association probability. In this manner, the related association probability is decreased through the preset adjustment parameter to obtain the corrected association probability, so that the corrected association probability may be determined rapidly and conveniently, and furthermore, the accuracy of determining the association between the body bounding box and the hand bounding box in the circumscribed box may be improved.

[ 0020] In some embodiments, the operation that the degree of association between the body bounding box and the hand bounding box that form the circumscribed box is determined based on the at least two target hand key points may include that: in a case that the at least two target hand key points include no target hand key point of which the confidence is greater than the preset threshold, the association probability between the body bounding box and the hand bounding box that form the circumscribed box is decreased to obtain the corrected association probability; and the corrected association probability is determined as the target association probability. In this manner, in a case that the target hand key points include no target hand key point of which the confidence is greater than the preset threshold, the association probability between the body bounding box and the hand bounding box that form the circumscribed box is decreased, so that the accuracy in association of the body bounding box and the hand bounding box that form the circumscribed box may further be improved.

[ 0021] In some embodiments, the target hand key point may include a wrist key point. A more accurate detection parameter may be provided, and furthermore, the detection accuracy may be improved.

[ 0022] In some embodiments, in a case of determining multiple body bounding boxes and/or multiple hand bounding boxes in the image to be detected, the method may further include that: each detected body bounding box is combined with each detected hand bounding box to obtain multiple bounding box pairs; the operation that the circumscribed box of the body bounding box and the hand bounding box is determined may include that: a circumscribed box of the body bounding box and the hand bounding box in each bounding box pair is determined; and the operation that the degree of association between the body and hand in the circumscribed box is determined based on the hand key point in the circumscribed box and the association probability may include that: a degree of association between a body and hand in each circumscribed box is determined to obtain a set of target degrees of association, a candidate degree of association greater than a preset association degree threshold is determined in the set of target degrees of association, and based on a target circumscribed box corresponding to the candidate degree of association, a hand matching a body in the target circumscribed box is determined. In this manner, a candidate degree of association consistent with a certain condition is screened based on a target degree of association corresponding to the circumscribed box formed by each body and each hand, and the hand matching each body is further determined, so that a more accurate body and hand matching relationship may be determined based on the optimized degree of association.

[ 0023] The embodiments of the disclosure provide a body and hand association apparatus, which may include a first acquisition module, a first determination module, a second determination module, a third determination module, and a fourth determination module.

[ 0024] The first acquisition module may be configured to acquire an image to be detected of which an image content includes a body and a hand.

[ 0025] The first determination module may be configured to determine a body bounding box of the body and a hand bounding box of the hand in the image to be detected.

[ 0026] The second determination module may be configured to determine an association probability between the body bounding box and the hand bounding box.

[ 0027] The third determination module may be configured to determine a circumscribed box of the body bounding box and the hand bounding box.

[ 0028] The fourth determination module may be configured to determine a degree of association between the body and hand in the circumscribed box based on a hand key point in the circumscribed box and the association probability.

[ 0029] In some embodiments, the second determination module may include: a feature determination sub-module, configured to determine a body feature of the body bounding box and a hand feature of the hand bounding box; and an association probability determination sub-module, configured to determine the association probability between the body bounding box and the hand bounding box based on the body feature and the hand feature.

[ 0030] In some embodiments, the third determination module may include: a bounding box determination sub-module, configured to determine, as a target bounding box pair, a body bounding box and a hand bounding box between which an association probability is greater than a preset probability threshold; and a circumscribed box determination sub-module, configured to determine a circumscribed box of the body bounding box and the hand bounding box in the target bounding box pair.

[ 0031] In some embodiments, the circumscribed box determination sub-module may include: a candidate box determination subunit, configured to determine a candidate box that encloses the target bounding box pair; and a circumscribed box determination subunit, configured to adjust a cover area of the candidate box based on a relative positional relationship between the body bounding box and the hand bounding box in the target bounding box pair to obtain the circumscribed box.

[ 0032] In some embodiments, the fourth determination module may include: a hand key point determination sub-module, configured to determine at least two target hand key points in the circumscribed box, the at least two target hand key points including key points that belong to two different hands of a same person respectively; and an association degree determination sub-module, configured to determine, based on the at least two target hand key points, the degree of association between the body bounding box and the hand bounding box that form the circumscribed box.

[ 0033] In some embodiments, the association degree determination sub-module may include: a confidence determination subunit, configured to determine a confidence that each target hand key point is a ground truth point; a distance determination subunit, configured to, in a case that any confidence is greater than a preset threshold, determine a distance from the target hand key point corresponding to the confidence greater than the preset threshold to the hand bounding box that forms the circumscribed box; an association probability determination subunit, configured to determine, based on the distance, a target association probability between the body bounding box and the hand bounding box that form the circumscribed box; and an association degree determination subunit, configured to determine, based on the target association probability, the degree of association between the body bounding box and the hand bounding box that form the circumscribed box.

[ 0034] In some embodiments, the association probability determination subunit is further configured to, in a case that the distance is less than or equal to a preset distance, determine the association probability between the body bounding box and the hand bounding box that form the circumscribed box to obtain an in-box association probability, and determine the in-box association probability as the target association probability.

[ 0035] In some embodiments, the association probability determination subunit is further configured to, in a case that the distance is greater than the preset distance, decrease the association probability between the body bounding box and the hand bounding box that form the circumscribed box to obtain a corrected association probability, and determine the corrected association probability as the target association probability.

[ 0036] In some embodiments, the association probability determination subunit is further configured to, in the case that the distance is greater than the preset distance, acquire a preset adjustment parameter configured to adjust the association probability, and lower the association probability between the body bounding box and the hand bounding box that form the circumscribed box using the preset adjustment parameter to obtain the corrected association probability.

[ 0037] In some embodiments, the association degree determination sub-module is further configured to, in a case that the at least two target hand key points include no target hand key point of which the confidence is greater than the preset threshold, decrease the association probability between the body bounding box and the hand bounding box that form the circumscribed box to obtain the corrected association probability, and determine the corrected association probability as the target association probability.

[ 0038] In some embodiments, the target hand key point includes a wrist key point.

[ 0039] In some embodiments, in a case of determining multiple body bounding boxes and/or multiple hand bounding boxes in the image to be detected, the bounding box determination sub-module is further configured to combine each detected body bounding box and each detected hand bounding box to obtain multiple bounding box pairs; the circumscribed box determination sub-module is further configured to determine a circumscribed box of the body bounding box and the hand bounding box in each bounding box pair; and the fourth determination module is further configured to determine a degree of association between a body and hand in each circumscribed box to obtain a set of target degrees of association, determine a candidate degree of association greater than a preset association degree threshold in the set of target degrees of association, and based on a target circumscribed box corresponding to the candidate degree of association, determine a hand matching a body in the target circumscribed box.

[ 0040] The embodiments of the disclosure provide a computer device, which may include a memory and a processor. Computer-executable instructions may be stored in the memory. The processor may run the computer-executable instructions in the memory to implement the body and hand association method.

[ 0041] Correspondingly, the embodiments of the disclosure provide a computer storage medium having stored thereon computer-executable instructions that, when executed, cause the body and hand association method to be implemented.

[ 0042] The embodiments of the disclosure provide the body and hand association method and apparatus, the device, and the storage medium. First, the body bounding box of the body and the hand bounding box of the hand are determined in the image to be detected of which the image content includes the body and the hand. Then, the association probability between the body bounding box and the hand bounding box is determined, and the circumscribed box of the body bounding box and the hand bounding box is determined. Finally, the degree of association between the body and hand in the circumscribed box is determined based on the hand key point in the circumscribed box and the association probability. In this manner, the degree of association between the body and hand in the circumscribed box is determined based on the association probability between the body bounding box and the hand bounding box and the hand key point in the circumscribed box of the body bounding box and the hand bounding box, so that the accuracy in association of the body bounding box and the hand bounding box in the circumscribed box may be improved.

BRIEF DESCRIPTION OF THE DRAWINGS

[ 0043] In order to describe the technical solutions of the embodiments of the disclosure more clearly, the drawings required to be used in the descriptions about the embodiments will be simply introduced below. It is apparent that the drawings described below are merely some embodiments of the disclosure. Other drawings may further be obtained by those of ordinary skill in the art according to these drawings without creative work. [ 0044] FIG. 1 is a flowchart of a body and hand association method according to an embodiment of the disclosure.

[ 0045] FIG. 2 is a flowchart of another body and hand association method according to an embodiment of the disclosure.

[ 0046] FIG. 3A is a flowchart of another body and hand association method according to an embodiment of the disclosure.

[ 0047] FIG. 3B is a flowchart of another body and hand association method according to an embodiment of the disclosure.

[ 0048] FIG. 4 is a schematic diagram of an application scene of a body and hand association method according to an embodiment of the disclosure.

[ 0049] FIG. 5 is a composition structure diagram of a body and hand association apparatus according to an embodiment of the disclosure.

[ 0050] FIG. 6 is a composition structure diagram of a computer device according to an embodiment of the disclosure.

DETAILED DESCRIPTION

[ 0051] In order to make the purposes, technical solutions, and advantages of the embodiments of the disclosure clearer, specific technical solutions of the disclosure will further be described below in combination with the drawings in the embodiments of the disclosure in detail. The following embodiments are used to describe the embodiments of the disclosure rather than limit the scope of the embodiments of the disclosure.

[ 0052] "Some embodiments" involved in the following descriptions describes a subset of all possible embodiments. However, it can be understood that "some embodiments" may be the same subset or different subsets of all the possible embodiments, and may be combined without conflicts.

[ 0053] Term "first/second/third" involved in the following descriptions is only for distinguishing similar objects, and does not represent a specific sequence of the objects. It can be understood that "first/second/third" may be interchanged to specific sequences or orders if allowed to implement the embodiments of the disclosure described herein in sequences except the illustrated or described ones.

[ 0054] Unless otherwise defined, all technological and scientific terms used in the disclosure have meanings the same as those usually understood by those skilled in the art of the embodiments of the disclosure. Terms used in the disclosure are only adopted to describe the embodiments of the disclosure and not intended to limit the embodiments of the disclosure.

[ 0055] Before the embodiments of the disclosure are further described in detail, nouns and terms involved in the embodiments of the disclosure will be described. The nouns and terms involved in the embodiments of the disclosure are suitable for the following explanations.

[ 0056] 1) Computer Vision (CV), as a science researching how to make machines

"see", refers to recognizing, tracking, and measuring targets using video cameras and computers instead of human eyes, and further performing image processing.

[ 0057] 2) Deep Learning (DL) is introduced to Machine Learning (ML) as a new research direction in the field of ML to make ML closer to the initial goal: Artificial Intelligence (Al). DL refers to learning inherent laws and representation layers of sample data. Information obtained in these learning processes helps greatly the interpretation of data such as texts, images, and sounds. The final goal of DL is to make machines able to analyze and learn like humans, and able to recognize data such as texts, images, and sounds.

[ 0058] 3) HKD, also called human pose estimation, is a relatively elementary task in

CV, as well as a predecessor task of human action recognition, behavior analysis, humancomputer interaction, etc. Under a normal condition, HKD may be subdivided into single-person/multi-person key point detection and two-dimensional/three-dimensional key point detection. In addition, in some algorithms, key point tracking, also called human pose tracking, may further be performed after key point detection is completed.

[ 0059] 4) Association algorithm is a kind of important algorithm in data mining.

Association reflects knowledge about a dependence or association between an event and another event.

[ 0060] 5) Confidence: in statistics, a confidence interval of a probability sample is an interval estimate of a certain population parameter of the sample. The confidence interval presents a degree that a true value of the parameter falls around a measurement result at a certain probability. The confidence interval defines a reliability range of the measured value of the measured parameter, i.e., the previously required "certain probability". This probability is called a confidence level.

[ 0061] An exemplary application of a body and hand association device provided in the embodiments of the disclosure will be described below. The device provided in the embodiments of the disclosure may be implemented as various types of user terminals with an image collection function, such as a notebook computer, a tablet computer, a desktop computer, a camera, and a mobile device (e.g., a personal digital assistant, a dedicated messaging device, and a portable game device), or may be implemented as a server. The exemplary application of the device implemented as the terminal or the server will be described below.

[ 0062] A method may be applied to a computer device. A function realized by the method may be realized by calling a program code through a processor in the computer device. Of course, the program code may be stored in a computer storage medium. It can be seen that the computer device at least includes the processor and the storage medium.

[ 0063] The embodiments of the disclosure provide a body and hand association method. As shown in FIG. 1, descriptions will be made in combination with operations shown in FIG. 1.

[ 0064] In S 101, an image to be detected of which an image content includes a body and a hand is acquired.

[ 0065] In some embodiments, the image to be detected may be acquired by a body and hand association apparatus through an internal image collection module. Alternatively, the image to be detected may be received from an apparatus or device capable of performing information interaction with the body and hand association apparatus. Correspondingly, the image to be detected may be a color image, or may be a gray image. The body and the hand may be in a foreground region, medium shot region, and background region in the image to be detected. A region where the body and hand are located in the image to be detected is not limited in the embodiment of the disclosure.

[ 0066] In some embodiments, the body in the image to be detected refers to a body of which all body information may be presented in the image to be detected, or a body of which part of body information is presented. In a case that all the body information of the body is presented in the image to be detected, such as the head, the two arms, the whole trunk, and the two legs, or part of the body information is presented, such as the head, one arm, and the two legs, the hand in the image to be detected may refer to left hand information, right hand information, left and right hand information, etc., presented in the image to be detected.

[ 0067] In some embodiments, pose information of the body in the image to be detected may be standing, walking, sitting, etc. Correspondingly, pose information of the hand in the image to be detected may be unclenched or clenched, or may be partially clenched, etc.

[ 0068] In some embodiments, the image content of the image to be detected includes, but not limited to, the body and the hand. The image content of the image to be detected may be that the body and hand are in any scene, such as a classroom, a park, an office, or a game place. In addition, the image content in the image to be detected may include one, two or more than two bodies and hands. Relative positional relationships between the bodies, between the hands, and between the bodies and the hands in the image content of the image to be detected may be left-right, front-back, one under the other, etc.

[ 0069] In S102, a body bounding box of the body and a hand bounding box of the hand are determined in the image to be detected.

[ 0070] In some embodiments, the body bounding boxy is a bounding boxy that may include a body part of the body in the image to be detected. Exemplarily, if all the body parts of the body may be presented in the image to be detected, the body bounding box is a minimum rectangular box that may enclose the head, two arms, whole trunk, two legs, etc., of the body. In addition, the body bounding box may enclose key points of, for example, the head, neck, shoulders, elbows, hands, hips, knees, and feet of the body. Exemplarily, if part of the body parts of the body may be presented in the image to be detected, the body bounding box is a minimum rectangular box that may enclose the head, one arm, partial trunk, etc., of the body.

[ 0071] In some embodiments, the hand bounding box of the hand may be a part of the hand presented in the image to be detected. Exemplarily, if the whole hand may be presented in the image to be detected, the hand bounding box is a minimum rectangular box that may enclose the fingertips, interphalangeal joints, etc., of the hand. If part of information of the hand may be presented in the image to be detected, the hand bounding box is a minimum rectangular box that may enclose part of fingertips, part of interphalangeal joints, etc., of the hand.

[ 0072] In some embodiments, a positional relationship between the body bounding box of the body and the hand bounding box of the hand may be partial overlapping or neighboring, or may even be that the hand bounding box is in the body bounding box, etc. In addition, a size of the body bounding box is determined based on a size, presented in the image to be detected, of the body. Correspondingly, a size of the hand bounding box is determined based on a size, presented in the image to be detected, of the hand.

[ 0073] In some embodiments, it is assumed that the image to be detected includes multiple bodies and multiple hands. Each body corresponds to a body bounding box, and there may be a relationship such as overlapping, partial overlapping, and neighboring between the body bounding boxes corresponding to different bodies. Each hand corresponds to a hand bounding box, and there may be a relationship such as overlapping, partial overlapping, and neighboring between the hand bounding boxes corresponding to different hands. In addition, the hand bounding boxes may be divided into two major types: i.e., left hand bounding boxes and right hand bounding boxes.

[ 0074] In some implementation modes, the body and hand in the image to be detected respectively through a trained detection model to determine the corresponding body bounding box and hand bounding box.

[ 0075] In some embodiments, the body bounding box and the hand bounding box may be generated based on a target detection algorithm. Exemplarily, in the embodiment of the disclosure, the target detection algorithm, such as a Fast Region Convolutional Neural Network (Fast R-CNN), a Faster R-CNN, and a Region based Fully Convolutional Neural Network (R-FCN), may be used to detect the body and hand in the image to be detected to obtain the body bounding box of the body and the hand bounding box of the hand. Target detection based on the Faster R-CNN may be implemented through the following four basic steps, such as candidate region generation, feature extraction, classification, and position refinement. As such, more accurate body bounding box and hand bounding box may be determined rapidly and conveniently. In addition, use of the target detection algorithm may reduce calculations.

[ 0076] In S103, an association probability between the body bounding box and the hand bounding box is determined.

[ 0077] In some embodiments, the association probability may refer to an association degree between the body bounding box and the hand bounding box, or an association possibility therebetween. Here, "association" refers to that there is a belonging relationship between physical objects represented by the two bounding boxes, for example, there is a belonging relationship between the hand and the body. The association degree or the association possibility may be determined based on color features, texture features, shape feature, spatial relationship features, etc., between the body bounding box and the hand bounding box. Exemplarily, in the embodiment of the disclosure, if the association probability between the body bounding box and the hand bounding box is relatively high, it indicates that an association degree between related feature information of the body bounding box and the hand bounding box is relatively high. Similarly, if the association probability between the body bounding box and the hand bounding box is relatively low, it indicates that the association degree between the related feature information of the body bounding box and the hand bounding box is relatively low.

[ 0078] In some embodiments, the association probability between the body bounding box and the hand bounding box may be determined through a related association algorithm. Correspondingly, the association probability may be represented as a decimal, a fraction, a percentage, etc. A numerical value of the association probability may be 90%, 0.75, etc.

[ 0079] In a possible implementation mode, a body feature of the body bounding box and a hand feature of the hand bounding box may be acquired to determine the association probability between the body bounding box and the hand bounding box, which may be implemented through the following process.

[ 0080] At block 1, a body feature of the body bounding box and a hand feature of the hand bounding box are determined.

[ 0081] In some embodiments, the body feature may refer to a color feature, texture feature, shape feature, and spatial relationship feature of the body in the body bounding box. Exemplarily, the body feature may refer to pose information, relative position information, color information, etc., of the body in the body bounding box. The hand feature may refer to a color feature, texture feature, shape feature, and spatial relationship feature of the hand in the hand bounding box. Exemplarily, the hand feature may refer to pose information, relative position information, color information, etc., of the hand in the hand bounding box.

[ 0082] In some embodiments, the body feature of the body bounding box and the hand feature of the hand bounding box may be extracted and determined through a feature extraction model. As such, a more accurate body feature in the body bounding box and a more accurate hand feature in the hand bounding box may be determined, and furthermore, the accuracy of determining an association between the body bounding box and the hand bounding box may be improved based on the relatively accurate body feature and hand feature.

[ 0083] At block 2, the association probability between the body bounding box and the hand bounding box is determined based on the body feature and the hand feature.

[ 0084] In some embodiments, the body feature and the hand feature are correlated through an association algorithm to determine the association probability between the body bounding box and the hand bounding box. Exemplarily, feature information matching may be performed on the body feature and the hand feature to further determine the association probability between the body bounding box and the hand bounding box. In this manner, information matching is performed on the body feature and the hand feature to determine the association probability between the body bounding box and the hand bounding box, so that the accuracy of determining the association between the body bounding box and the hand bounding box may be improved.

[ 0085] In some possible embodiments, feature extraction is performed on the body bounding box and the hand bounding box using the feature extraction model respectively to obtain the corresponding body feature and hand feature. As such, the body feature and the hand feature may be extracted rapidly and accurately to further determine the association probability between the body bounding box and the hand bounding box based on the extracted body feature and hand feature, and furthermore, the accuracy of determining the association between the body bounding box and the hand bounding box may be improved.

[ 0086] In S104, a circumscribed box of the body bounding box and the hand bounding box is determined.

[ 0087] In some embodiments, the circumscribed box may refer to a minimum rectangular box that may enclose the body bounding box and the hand bounding box, or may refer to a minimum rectangular box that is formed by a body bounding box and a hand bounding box between which an association probability is greater than a preset probability threshold and that may enclose the body bounding box and the hand bounding box. Exemplarily, the circumscribed box may be determined based on the body bounding box, the hand bounding box, and a relative positional relationship between the body bounding box and the hand bounding box. Information in the circumscribed box includes, but not limited to, feature information of the body bounding box and feature information of the hand bounding box. In addition, the hand bounding box in the circumscribed box may be at any position in the circumscribed box.

[ 0088] In some embodiments, the hand bounding box includes a right hand bounding box and a left hand bounding box. The body bounding box may form two circumscribed boxes with the right hand bounding box and the left hand bounding box respectively. The two circumscribed boxes may positionally overlap, or completely overlap, etc.

[ 0089] In some embodiments, in a case that there are multiple body bounding boxes and multiple hand bounding boxes, each body bounding box and each hand bounding box form a circumscribed box. A size of the circumscribed box is determined by the sizes of the body bounding box and the hand bounding box. In addition, positional relationships between the circumscribed boxes in the image to be detected are determined by the body bounding boxes and the hand bounding boxes.

[ 0090] In S105, a degree of association between the body and hand in the circumscribed box is determined based on a hand key point in the circumscribed box and the association probability.

[ 0091] In some embodiments, the hand key point in the circumscribed box may refer to hand palm center information or hand wrist key point information in the circumscribed box. Exemplarily, it is assumed that the hand key point refers to the hand wrist key point information, i.e., information of a wrist joint of the hand. The hand wrist key point information may include a confidence that a hand wrist key point is a ground truth point and a distance from the hand wrist key point to the hand bounding box that forms the circumscribed box. For example, a confidence that a left wrist key point is a ground truth point, a confidence that a right wrist key point is a ground truth point, a distance from a right wrist key point to the hand bounding box that forms the circumscribed box, and a distance from a left wrist key point to the hand bounding box that forms the circumscribed box are included. The hand wrist refers to part information, that connects the palm with the forearm, of the body.

[ 0092] In some embodiments, the degree of association may refer to an association degree between the body, i.e., the body bounding box, and hand, i.e., the hand bounding box, in the circumscribed box. The association degree may be represented by a numerical value, such as 0 to 100. Exemplarily, if the body bounding box is uncorrelated with the hand bounding box, the numerical value corresponding to the association degree is 0. If the body bounding box is highly correlated with the hand bounding box, the numerical value corresponding to the association degree may be represented as 80 to 100.

[ 0093] In some embodiments, the confidence that the hand key point in the circumscribed box is a ground truth point and the distance from the hand key point to the hand bounding box that forms the circumscribed box are judged to determine the degree of association between the body and hand in the circumscribed box. Exemplarily, the degree of association between the body and hand in the circumscribed box may also be represented as matched and mismatched, or highly correlated or generally correlated.

[ 0094] According to the body and hand association method provided in the embodiment of the disclosure, first, the body bounding box of the body and the hand bounding box of the hand are determined in the image to be detected of which the image content includes the body and the hand. Then, the association probability between the body bounding box and the hand bounding box is determined, and the circumscribed box of the body bounding box and the hand bounding box is determined. Finally, the degree of association between the body and hand in the circumscribed box is determined based on the hand key point in the circumscribed box and the association probability. In this manner, the degree of association between the body and hand in the circumscribed box is determined based on the association probability between the body bounding box and the hand bounding box and the hand key point in the circumscribed box of the body bounding box and the hand bounding box, so that the accuracy in association of the body bounding box and the hand bounding box in the circumscribed box may be improved.

[ 0095] In a possible implementation mode, a body bounding box and a hand bounding box between which an association probability is greater than the preset probability threshold form a target bounding box pair, and meanwhile, a candidate that may enclose the target bounding box pair is determined as a circumscribed box of the body bounding box and the hand bounding box. That is, S104 may be implemented through the following process. As shown in FIG. 2, FIG. 2 is a flowchart of another body and hand association method according to an embodiment of the disclosure. The following descriptions will be made in combination with the operations shown in FIG. 1 and FIG. 2. [ 0096] In S 141, a body bounding box and a hand bounding box between which an association probability is greater than a preset probability threshold are determined as a target bounding box pair.

[ 0097] In some embodiments, the preset probability threshold is determined based on practical applications, or may be determined based on attribute information of the image to be detected, or may be determined based on the body feature of the body bounding box and the hand feature of the hand bounding box. The preset probability threshold may be 90%, 85%, 88%, etc. Correspondingly, the preset probability threshold may be represented as a fraction, a decimal, and a percentage. Each target bounding box pair includes a body bounding box and a hand bounding box, and an association probability between the body bounding box and the hand bounding box is greater than the preset probability threshold.

[ 0098] In S142, a circumscribed box of the body bounding box and the hand bounding box in the target bounding box pair is determined.

[ 0099] In some embodiments, the circumscribed box may refer to a minimum rectangular box that may completely enclose the body bounding box and the hand bounding box in the target bounding box pair. In a case that the preset probability threshold is 90%, the association probability between the body bounding box and the hand bounding box in the circumscribed box is greater than 90%.

[ 00100] In some embodiments, the target bounding box pair formed by the body bounding box and the hand bounding box is screened based on the association probability to further obtain the circumscribed box of the body bounding box and the hand bounding box in the target bounding box pair. As such, a circumscribed box of a body bounding box and a hand bounding box between which a degree of association is relatively high may be determined efficiently and accurately.

[ 00101] In some possible implementation modes, the operation that the circumscribed box of the body bounding box and the hand bounding box in the target bounding box pair is determined, i.e., S142, may be implemented through the following process. The following descriptions will be continued in combination with the operations shown in FIG. 1 and FIG. 2.

[ 00102] At block 1, a candidate box that encloses the target bounding box pair is determined.

[ 00103] In some embodiments, in a case that the association probability is greater than the preset probability threshold, the body bounding box and the hand bounding box are combined based on a relative positional relationship between the body bounding box and the hand bounding box to obtain the candidate box that may enclose the target bounding box pair, i.e., the candidate box that may enclose the body bounding box and the hand bounding box. A size and shape of the candidate box are not limited in the embodiment of the disclosure.

[ 00104] In some embodiments, in a case that multiple body bounding boxes and multiple hand bounding boxes are determined, and association probabilities are greater than the preset probability threshold, a first body bounding box is combined with a first right hand bounding box to obtain a first target bounding box pair, and a first candidate box that may enclose the first target bounding box pair is determined. In some possible implementation modes, the first body bounding box is combined with a second right hand bounding box to obtain a second target bounding box pair, and a second candidate box that may enclose the second target bounding box pair is determined. A second body bounding box is combined with a third right hand bounding box to obtain a third target bounding box pair, and a third candidate box that may enclose the third target bounding box pair, etc., are determined. The second body bounding box is combined with a second left hand bounding box to obtain a fourth target bounding box pair, and a fourth candidate box that may enclose the fourth target bounding box pair, etc., are determined.

[ 00105] At block 2, a cover area of the candidate box is adjusted based on a relative positional relationship between the body bounding box and the hand bounding box in the target bounding box pair to obtain the circumscribed box.

[ 00106] In some embodiments, the cover area of the candidate box is adjusted based on the relative positional relationship between the body bounding box and the hand bounding box in the target bounding box pair to obtain the circumscribed box. The circumscribed box is a minimum rectangular box that may enclose the body bounding box and the hand bounding box. Exemplarily, adjusting the cover area of the candidate box includes, but not limited to, correspondingly reducing the cover area of the candidate box according to the shape of the candidate box, reducing the cover area according to body information in the candidate box, etc. As such, a more accurate circumscribed box may be determined rapidly and efficiently to improve the accuracy of determining the association between the body bounding box and the hand bounding box that form the circumscribed box based on the hand key point in the more accurate circumscribed box.

[ 00107] Based on the above descriptions, it can be seen that the association probability between the body bounding box and the hand bounding box in the circumscribed box is greater than the preset probability threshold. Therefore, there is a certain association between the body and hand in the circumscribed box, namely there may not be any circumscribed box where a numerical value corresponding to an association degree between a body bounding box and a hand bounding box is 0.

[ 00108] In some possible implementation modes, at least two target hand key points in the circumscribed box and the association probability are determined to determine the degree of association between the body bounding box and the hand bounding box that form the circumscribed box. That is, S105 may be implemented through the following process. The following descriptions will be continued in combination with the operations shown in FIG. 1 and FIG. 2.

[ 00109] In S151, at least two target hand key points in the circumscribed box are determined.

[ 00110] The at least two target hand key points include key points that belong to two different hands of a same person respectively.

[ 00111] In some embodiments, the at least two target hand key points in the circumscribed box may refer to the key points that belong to the two different hands of the same person respectively in the circumscribed box. The key points that belong to the two different hands of the same person respectively may be at any positions in the circumscribed box. As such, the at least two target hand key points in the circumscribed box may be determined to reduce errors possibly generated when the degree of association is judged by detecting only one hand key point.

[ 00112] In some embodiments, the at least two target hand key points in the circumscribed box may be determined through an HKD model.

[ 00113] In a possible implementation mode, the target hand key point includes a wrist key point.

[ 00114] In some embodiments, in a case that the target hand key point includes the wrist key point, the at least two target hand key points in the circumscribed box may be wrist key points that belong to the two different hands of the same person respectively in the circumscribed box. In the following embodiments of the disclosure, the target hand key point refers to the wrist key point.

[ 00115] In some embodiments, the hand key point is determined to be the wrist key point. As such, a more accurate detection parameter may be provided, and furthermore, the detection accuracy may be improved.

[ 00116] In S152, the degree of association between the body bounding box and the hand bounding box that form the circumscribed box is determined based on the at least two target hand key points.

[ 00117] In some embodiments, the degree of association between the body bounding box and the hand bounding box that form the circumscribed box is determined based on key points of different wrists, such as a key point of a right wrist and a key point of a left wrist. In this manner, on the premise that whether the hand bounding box that forms the circumscribed box is a left hand bounding box or a right hand bounding box cannot be determined, the key point of the left wrist and the key point of the right wrist are simultaneously detected, so that errors possibly generated when the degree of association is judged by detecting the key point of only one wrist may be reduced, namely the key points of the wrists in the circumscribed box may be comprehensively detected to further improve the accuracy in association of the body bounding box and the hand bounding box that form the circumscribed box.

[ 00118] In some embodiments, the operation that the degree of association between the body bounding box and the hand bounding box that form the circumscribed box is determined based on the at least two target hand key points, i.e., the key points of different wrists, i.e., S152, may be implemented through the following operations. As shown in FIG. 3, FIG. 3 is a flowchart of another body and hand association method according to an embodiment of the disclosure. The following descriptions will be made in combination with the operations shown in FIG. 1 to FIG. 3.

[ 00119] In S201, a confidence that each target hand key point is a ground truth point is determined.

[ 00120] In some embodiments, a confidence that each wrist key point is a ground truth point is determined. The ground truth point may refer to information that each wrist key point in the circumscribed box is ground truth, i.e., a true value. Exemplarily, it may indicate that, under a certain condition, the ground truth point is an objective true value after measurement in the circumscribed box. The confidence that each wrist key point is a ground truth point may be acquired from the image to be detected through the image feature extraction model or the target detection algorithm.

[ 00121] In some embodiments, the confidence that each wrist key point is a ground truth point may refer to a confidence level that each wrist key point is a ground truth point. Exemplarily, if a confidence interval at a confidence level 0.95 that the right wrist key point is a ground truth point is (0.5, 0.6), a confidence that the right wrist key point is a ground truth point is 50% to 60% at a probability of 95%.

[ 00122] In some embodiments, the confidence that each target hand key point in the circumscribed box is a ground truth point may refer to a confidence that the left wrist key point is a ground truth point and the confidence that the right wrist key point is a ground truth point.

[ 00123] In a possible implementation mode, the confidence that the target hand key point is a ground truth point is judged to determine the target association probability between the body bounding box and the hand bounding box that form the circumscribed box. There may be two conditions. [ 00124] In a case that any confidence is greater than a preset threshold, following S202,

5203 and S206 are executed. The following descriptions will be made in combination with the operations in FIG. 1, FIG. 2, and FIG. 3 A.

[ 00125] In a case that not confidence is greater than the preset threshold, following

5204 to S206 are executed. The following descriptions will be made in combination with the operations in FIG. 1, FIG. 2, and FIG. 3B.

[ 00126] In S202, in a case that any confidence is greater than a preset threshold, a distance from the target hand key point corresponding to the confidence greater than the preset threshold to the hand bounding box that forms the circumscribed box is determined.

[ 00127] In some embodiments, a representation form of the preset threshold forms a mapping relationship with a representation form of the confidence. The preset threshold may be set according to requirements on a resolution and/or recognizability of the wrist key point. For example, if the requirements on the resolution and recognizability of the wrist key point are higher, the preset threshold is set to be greater. Exemplarily, the preset threshold may be set to 0.85. In a case that the confidence is greater than the preset threshold, the distance from the wrist key point corresponding to the confidence greater than the preset threshold to the hand bounding box that forms the circumscribed box is determined. The distance may be a distance from each wrist key point to a center point of the hand bounding box that forms the circumscribed box, or may be a distance from each wrist key point to any joint of the hand in the hand bounding box that forms the circumscribed box, or may be a distance from each wrist key point to a center point of the hand in the hand bounding box that forms the circumscribed box, or may be a distance from each wrist key point to a boundary region of the hand bounding box that forms the circumscribed box.

[ 00128] In a possible implementation mode, first distances and second distances from different wrist key points to the hand bounding box that forms the circumscribed box may be determined to further determine the degree of association between the body bounding box and the hand bounding box that form the circumscribed box based on the first distances and/or the second distances. That is, in a case that different wrist key points include a first key point of a first wrist and a second key point of a second wrist, and the distance includes a first distance and a second distance, S202 may be implemented through the following process.

[ 00129] At block 1, in a case that a confidence corresponding to the first key point is greater than the preset threshold, a first distance from the first key point to the hand bounding box that forms the circumscribed box is determined.

[ 00130] In some embodiments, in a case that the confidence corresponding to the first key point of the first wrist is greater than the preset threshold, the first distance from the first key point to the hand bounding box that forms the circumscribed box is determined. The first wrist may refer to any one of the left and right wrists of the body in the circumscribed box. It is assumed that the first wrist is the left wrist, and the second wrist is correspondingly the right wrist. It is assumed that the first wrist is the right wrist, and the second wrist is correspondingly the left wrist.

[ 00131] In some embodiments, if the first wrist refers to the right wrist, in a case that the confidence corresponding to the first key point of the right wrist is greater than the preset threshold, it may be determined that the first distance is a distance from the first key point of the right wrist to the boundary region of the hand bounding box that forms the circumscribed box, or a distance from the first key point of the right wrist to the center point of the hand bounding box that forms the circumscribed box, or a distance from the first key point of the right wrist to any joint of the hand in the hand bounding box that forms the circumscribed box, or the distance from the first key point of the right wrist to the boundary region of the hand bounding box that forms the circumscribed box, etc.

[ 00132] At block 2, in a case that a confidence corresponding to the second key point is greater than the preset threshold, a second distance from the second key point to the hand bounding box that forms the circumscribed box is determined.

[ 00133] In some embodiments, in a case that the confidence corresponding to the second key point of the second wrist is greater than the preset threshold, the second distance from the second key point to the hand bounding box that forms the circumscribed box is determined. Exemplarily, if the second wrist is the left wrist, the second distance may be a distance from the second key point of the left wrist to the boundary region of the hand bounding box that forms the circumscribed box, or a distance from the second key point of the left wrist to the center point of the hand bounding box that forms the circumscribed box, or a distance from the second key point of the left wrist to any joint of the hand in the hand bounding box that forms the circumscribed box, or the distance from the second key point of the left wrist to the boundary region of the hand bounding box that forms the circumscribed box, etc. The first distance may be the same as or different from the second distance.

[ 00134] In some embodiments, the confidences and related distances corresponding to the first key point of the first wrist and the second key point of the second wrist are judged respectively, so that the accuracy of determining the target association probability based on the distance may further be improved on the premise of providing a more accurate distance reference parameter.

[ 00135] In S203, a target association probability between the body bounding box and the hand bounding box that form the circumscribed box is determined based on the distance.

[ 00136] In some embodiments, the target association probability between the body bounding box and the hand bounding box that form the circumscribed box is determined based on the distance from each wrist key point to the hand bounding box that forms the circumscribed box. The target association probability may be the same as or different from the determined association probability between the body bounding box and the hand bounding box that form the circumscribed box.

[ 00137] In some embodiments, adjusting, based on the distance, the association probability between the body bounding box and the hand bounding box that form the circumscribed box may be increasing, decreasing, or keeping the association probability between the body bounding box and the hand bounding box that form the circumscribed box, to further obtain the target association probability.

[ 00138] In a possible implementation mode, the distance is judged to determine the target association probability between the body bounding box and the hand bounding box of the circumscribed box. That is, implementation of S203 may be divided into two conditions based on the distance.

[ 00139] A first condition may be implemented through following S231 and S232.

[ 00140] In S231, in a case that the distance is less than or equal to a preset distance, the association probability between the body bounding box and the hand bounding box that form the circumscribed box is determined to obtain an in-box association probability.

[ 00141] In some embodiments, a numerical value of the distance is judged, and in a case that the distance is less than or equal to the preset distance, the association probability between the body bounding box and the hand bounding box that form the circumscribed box is calculated to obtain the corresponding in-box association probability. Exemplarily, the in-box association probability may be an original association probability between the body bounding box and the hand bounding box that form the circumscribed box. Alternatively, the in-box association probability may be obtained by adaptively increasing the original association probability between the body bounding box and the hand bounding box that form the circumscribed box.

[ 00142] In S232, the in-box association probability is determined as the target association probability.

[ 00143] In some embodiments, determining the in-box association probability as the target association probability may be directly determining the in-box association probability as the target association probability. Exemplarily, it is assumed that the inbox association probability is the original association probability between the body bounding box and the hand bounding box that form the circumscribed box, namely the in-box association probability is greater than the preset probability threshold. Alternatively, the in-box association probability may be adaptively adjusted, for example, the in-box association probability is increased, and the increased in-box association probability is determined as the target association probability. In this manner, the numerical value of the distance may be judged to determine the target association probability between the body bounding box and the hand bounding box rapidly and conveniently, and meanwhile, unnecessary calculations may be reduced.

[ 00144] A second condition may be implemented through following S233 and S234.

[ 00145] In S233, in a case that the distance is greater than the preset distance, the association probability between the body bounding box and the hand bounding box that form the circumscribed box is decreased to obtain a corrected association probability.

[ 00146] In some embodiments, the numerical value of the distance is judged, and in a case that the distance is greater than the preset distance, the association probability between the body bounding box and the hand bounding box that form the circumscribed box is decreased. Exemplarily, in a case that the distance is greater than the preset distance, the association probability between the body bounding box and the hand bounding box that form the circumscribed box is lowered to obtain the corrected association probability. The original association probability between the body bounding box and the hand bounding box that form the circumscribed box may be decreased, namely the corrected association probability may be the same as, or less than, or greater than a preset association probability.

[ 00147] In some embodiments, a parameter that correction of the association probability between the body bounding box and the hand bounding box that form the circumscribed box depends on may be constant, or may be dynamically changed based on an absolute difference between the distance and the preset distance.

[ 00148] In a possible implementation mode, the association probability between the body bounding box and the hand bounding box that form the circumscribed box may be corrected through a preset adjustment parameter. That is, S233 may be implemented through the following operations.

[ 00149] At block 1, in the case that the distance is greater than the preset distance, a preset adjustment parameter configured to adjust the association probability is acquired.

[ 00150] In some embodiments, in a case that the distance is greater than the preset distance, the preset adjustment parameter configured to adjust the association probability is acquired. The preset adjustment parameter may be a constant, and a corresponding representation form may be a decimal, a fraction, a percentage, etc. The numerical representation form corresponding to the preset adjustment parameter forms corresponding relationship with the numerical representation form corresponding to the association probability. The preset adjustment parameter may be 1, 0.4, 5%, etc.

[ 00151] In some embodiments, the preset adjustment parameter configured to correct the association probability may be determined based on the distances from the left wrist key point and/or the right wrist key point to the hand bounding box that forms the circumscribed box, or may be determined based on attribute information of the circumscribed box, or may be determined based on a size parameter of the hand bounding box that forms the circumscribed box, or may be determined based on the size of the hand in the hand bounding box that forms the circumscribed box, the size of the body in the body bounding box that forms the circumscribed box, etc.

[ 00152] At block 2, the association probability between the body bounding box and the hand bounding box that form the circumscribed box is lowered using the preset adjustment parameter to obtain the corrected association probability.

[ 00153] In some embodiments, the association probability between the body bounding box and the hand bounding box that form the circumscribed box is lowered using the preset adjustment parameter. Exemplarily, the preset adjustment parameter may be subtracted from the association probability between the body bounding box and the hand bounding box that form the circumscribed box. The preset adjustment parameter may be any positive number. Alternatively, the association probability between the body bounding box and the hand bounding box that form the circumscribed box may be divided by the preset adjustment parameter to obtain the corrected association probability. The preset adjustment parameter may be any positive integer.

[ 00154] The corrected association probability is lower than the original association probability between the body bounding box and the hand bounding box that form the circumscribed box. The corrected association probability may be greater than the preset association probability, or may be less than or equal to the preset association probability. In this manner, the related association probability is decreased through the preset adjustment parameter to obtain the corrected association probability, so that the corrected association probability may be determined rapidly and conveniently, and furthermore, the accuracy of determining the association between the body bounding box and the hand bounding box in the circumscribed box may be improved.

[ 00155] In S234, the corrected association probability is determined as the target association probability.

[ 00156] In some embodiments, the corrected association probability may be directly determined as the target association probability. In this manner, the numerical value of the distance may be judged to adjust the related parameter rapidly and conveniently to determine the target association probability between the body bounding box and the hand bounding box, and meanwhile, unnecessary calculations may be reduced.

[ 00157] In S204, in a case that the at least two target hand key points include no target hand key point of which the confidence is greater than the preset threshold, the association probability between the body bounding box and the hand bounding box that form the circumscribed box is decreased to obtain the corrected association probability.

[ 00158] In some embodiments, in a case that the right wrist key point and the left wrist key point include no key point of which the confidence is greater than the preset threshold, the association probability between the body bounding box and the hand bounding box that form the circumscribed box is lowered or decreased to obtain the corrected association probability.

[ 00159] In S205, the corrected association probability is determined as the target association probability.

[ 00160] In some embodiments, in a case of determining that the target hand key points include no target hand key point of which the confidence is greater than the preset threshold, the association probability between the body bounding box and the hand bounding box that form the circumscribed box is decreased, so that the accuracy in association of the body bounding box and the hand bounding box that form the circumscribed box may further be improved.

[ 00161] In S206, the degree of association between the body bounding box and the hand bounding box that form the circumscribed box is determined based on the target association probability.

[ 00162] In some embodiments, the degree of association between the body bounding box and the hand bounding box that form the circumscribed box is determined based on the target association probability, namely based on the uncorrected in-box association probability or the corrected association probability. For example, if it is determined that the target association probability is 80%, the corresponding degree of association between the body bounding box and the hand bounding box that form the circumscribed box is relatively high. If it is determined that the target association probability is 60%, the corresponding degree of association between the body bounding box and the hand bounding box that form the circumscribed box is relatively low. If it is determined that the target association probability is 90%, the corresponding degree of association between the body bounding box and the hand bounding box that form the circumscribed box is quite high. In this manner, if the confidence that the wrist key point is a ground truth point is greater than the preset threshold, the association probability between the body bounding box and the hand bounding box that form the circumscribed box is adaptively adjusted based on the distance from the wrist key point to the hand bounding box that forms the circumscribed box, or if the confidence is less than or equal to the preset threshold, the association probability between the body bounding box and the hand bounding box that form the circumscribed box is decreased, so that the accuracy in association of the body bounding box and the hand bounding box that form the circumscribed box may further be improved.

[ 00163] In a possible implementation mode, in a case of determining multiple body bounding boxes and/or multiple hand bounding boxes in the image to be detected, in the body and hand association method, a matching relationship between the body and hand in the related circumscribed box may further be determined through the following process.

[ 00164] At block 1, each detected body bounding box is combined with each detected hand bounding box to obtain multiple bounding box pairs.

[ 00165] In some embodiments, it is assumed that two body bounding boxes and three hand bounding boxes are determined in the image to be detected, and six bounding box pairs are obtained based on each detected body bounding box and each detected hand bounding box, such as body bounding box 1-hand bounding box 1, body bounding box 1- hand bounding box 2, body bounding box 1-hand bounding box 3, body bounding box 2- hand bounding box 1, body bounding box 2-hand bounding box 2, and body bounding box 2-hand bounding box 3.

[ 00166] At block 2, a circumscribed box of the body bounding box and the hand bounding box in each bounding box pair is determined.

[ 00167] In some embodiments, block 2 corresponds to S104 provided in the body and hand association method, namely the circumscribed boxes of the body bounding boxes and the hand bounding boxes are determined.

[ 00168] In some embodiments, the circumscribed box of the body bounding box and each hand bounding box in each bounding box pair is determined. Each circumscribed box may be obtained by combining the body bounding box of each body and the hand bounding box of each hand. Exemplarily, the body bounding box of each body and the hand bounding box of each hand may be determined at first. Then, an association probability between each body bounding box and each hand bounding box is calculated. In a case that the association probability is greater than the preset probability threshold, the body bounding box and the hand bounding box consistent with a certain condition are combined to form the circumscribed box.

[ 00169] In some embodiments, a specific size and shape of the circumscribed box of each body bounding box and each hand bounding box may be determined based on the body bounding box of each body and the hand bounding box of each hand.

[ 00170] In some embodiments, each body may correspond to multiple circumscribed boxes. Exemplarily, each body bounding box at least forms two corresponding circumscribed boxes with two hand bounding boxes, such as the right hand bounding box and the right hand bounding box. In addition, each hand bounding box may correspond to one, two or more than two circumscribed boxes, etc.

[ 00171] After block 1 and block 2 are executed, S105 provided in the body and hand association method, i.e., the following operation, may be executed.

[ 00172] At block 3, a degree of association between a body and hand in each circumscribed box is determined to obtain a set of target degrees of association.

[ 00173] In some embodiments, the degree of association between the body and hand in each circumscribed box is calculated and determined. The target degree of association may be determined based on different hand key points in each circumscribed box, or may be determined based on different wrist key points in each circumscribed box. Exemplarily, the target degree of association may be determined based on a confidence that each wrist key point in each circumscribed box is a ground truth point and a distance from each wrist key point to the hand bounding box that forms the circumscribed box. Details of a related implementation solution may refer to the descriptions about determining the target association probability of the circumscribed box in the abovementioned embodiments, and elaborations are omitted herein.

[ 00174] At block 4, a candidate degree of association greater than a preset association degree threshold is determined in the set of target degrees of association.

[ 00175] In some embodiments, screening the degree of associations in the set of target degrees of association based on the preset association degree threshold may be screening the degree of associations in the set of target degrees of association based on the preset association degree threshold to determine the corresponding candidate degree of association, or may be screening the degree of associations in the set of target degrees of association based on a parameter less than the preset association degree threshold to determine the corresponding candidate degree of association.

[ 00176] At block 5, based on a target circumscribed box corresponding to the candidate degree of association, a hand matching a body in the target circumscribed box is determined.

[ 00177] In some embodiments, the hand is determined based on the target circumscribed box corresponding to the candidate degree of association. The target circumscribed box may include two circumscribed boxes formed by each body and two corresponding hand bounding boxes, such as the left hand bounding box and the right hand bounding box, or may include multiple circumscribed boxes formed by each body and multiple corresponding hand bounding boxes, such as multiple left hand bounding boxes and/or multiple right hand bounding boxes. In this manner, a candidate degree of association consistent with a certain condition is screened based on the target degree of association corresponding to the circumscribed box formed by each body and each hand, and the hand matching each body is further determined, so that a more accurate body and hand matching relationship may be determined based on the optimized degree of association.

[ 00178] In some embodiments, first body A corresponds to three target circumscribed boxes, i.e., circumscribed box Al formed by first body A and a first hand, corresponding to a candidate degree of association of 90%, circumscribed box A2 formed by first body A and a second hand, corresponding to a candidate degree of association of 99%, and circumscribed box A3 formed by the first body and a third hand, corresponding to a candidate degree of association of 70% respectively. Second body B corresponds to four target circumscribed boxes, i.e., circumscribed box Bl formed by second body B and the first hand, corresponding to a candidate degree of association of 60%, circumscribed box B2 formed by second body B and the second hand, corresponding to a candidate degree of association of 70%, circumscribed box B3 formed by second body B and a fourth hand, corresponding to a candidate degree of association of 99%, and circumscribed box B4 formed by second body B and a fifth hand, corresponding to a candidate degree of association of 95%. In the embodiment of the disclosure, the above parameters are calculated based on a greedy algorithm, a Kuhn and Munkres (KM) algorithm, etc., to determine the hand matching each body. For example, a calculation result may be that the hand matching first body A is the second hand, and the hand matching second body B is the fourth hand and the fifth hand.

[ 00179] The body and hand association method will be described below in combination with a specific embodiment. However, it is to be noted that the specific embodiment is only for describing the embodiments of the disclosure better and does not form improper limits to the embodiments of the disclosure.

[ 00180] In a complex scene with many people, such as a game place, there is usually such a phenomenon that persons are occluded by one another or arms are crossed. In addition, there are many bodies in a field of view, and a crossing condition of corresponding arms is usually relatively complex. In the related art, a body and hand detection and association algorithm is used to predict an association probability between each pair of body and hand, or HKD is used to detect a hand key point of each body, and meanwhile, another sensor is combined to judge a person that a hand belongs to. It is difficult for the body and hand detection and association algorithm to deal with conditions that hands are stretched out, arms are crossed, etc. An HKD model corresponding to the body and hand detection and association algorithm depends much on the accuracy of a body bounding box. If the body bounding box is inaccurate, and does not include any hand, the HKD model may output an incorrect result. Moreover, when HKD is used to detect a hand key point of each body, a position and size of the whole hand cannot be obtained, and meanwhile, the other sensor is required to be combined to judge the person that the hand belongs to, so the judgment accuracy cannot be ensured, and the judgment cost is relatively high. [ 00181] Based on this, the embodiment of the disclosure provides a body and hand association method. In a game place, a more accurate body and hand matching relationship may be determined through the body and hand association method provided in the embodiment of the disclosure. As shown in FIG. 4, a schematic diagram of an application scene of a body and hand association method according to an embodiment of the disclosure is shown. A method for correlating each body with each hand in FIG. 4 may be implemented through the following operations.

[ 00182] At block 1, all body bounding boxes and hand bounding boxes in FIG. 4 are predicted using a detection model. As shown in FIG. 4, multiple body bounding boxes with face information are presented. Correspondingly, the body bounding box is body, and the hand bounding box is hand. In addition, a face bounding box is face, and a card bounding box is chip. Exemplarily, 401, 402, 403, and 404 are bodies in an image to be detected respectively, and 405, 406, and 407 are hands.

[ 00183] At block 2, an association probability between the body bounding box of each body and the hand bounding box of each hand, which are determined in block 1, is calculated.

[ 00184] At block 3, a circumscribed box of a body bounding box and a hand bounding box between which the association probability is greater than a preset probability threshold is determined.

[ 00185] At block 4, the obtained circumscribed box is input to ah HKD model, and is detected to obtain information of at least two target hand key points in each circumscribed box, such as a confidence that each target hand key point is a ground truth point, and a distance from the target hand key point corresponding to a confidence greater than a preset threshold to the hand bounding box that forms the circumscribed box.

[ 00186] At block 5, taking a circumscribed box as an example for description, through a wrist key point in the circumscribed box, the wrist key point being a left wrist key point and right wrist key point that belong to the same person respectively, if confidences that the right wrist key point and the left wrist key point are ground truth points are greater than the preset threshold, namely the confidence corresponding to any one of x and y is greater than a certain preset threshold thres_kp, x and y representing a right wrist that is a ground truth point and a left wrist that is a ground truth point respectively, distances from x and y to the hand bounding box that forms the circumscribed box are determined. In a case that the distances are less than or equal to a preset distance, an uncorrected association probability is obtained without correcting an association probability between the body bounding box and the hand bounding box that form the circumscribed box. In a case that the distances are greater than the preset distance, the association probability between the body bounding box and the hand bounding box that form the circumscribed box is lowered to obtain a lowered association probability. In addition, when the confidence corresponding to any one of x and y is not greater than the certain preset threshold thres_kp, the association probability between the body bounding box and the hand bounding box that form the circumscribed box is lowered to obtain the lowered association probability. Moreover, as shown in FIG. 4, a confidence parameter corresponding to each bounding box is determined, such as face 1.00, hand 1.00, and hand 0.97.

[ 00187] At block 6, the body bounding boxes and hand bounding boxes in the circumscribed boxes corresponding to the obtained lowered association probability and uncorrected association probability are matched through a greedy algorithm or a KM algorithm to obtain the hand corresponding to each body.

[ 00188] In this manner, in the game place, a degree of association between the body and hand that form the circumscribed box is determined based on the hand key point in the circumscribed box and the association probability between the body bounding box and the hand bounding box that form the circumscribed box, so that the hand and body accuracy in association may be improved greatly.

[ 00189] In a specific example, the image to be detected is an image collected in the game place. The body is a player body, and the hand is a player hand. Through block 1 to block 6, first, an association probability between a body bounding box of each player body and a hand bounding box of each player hand is determined. Then, in a case that the association probability is greater than a preset probability threshold, the body bounding box of the player body is combined with the hand bounding box of the player hand to generate a circumscribed box. Next, through a hand key point in the circumscribed box, for example, a confidence that a wrist key point is a ground truth point is compared with a preset threshold. In a case that the confidence is greater than the preset threshold, a distance from each wrist key point to the hand bounding box of the circumscribed box is determined, and the association probability between the body bounding box and the hand bounding box in the circumscribed box is adaptively adjusted based on the distance. Alternatively, in a case that the confidence is less than or equal to the preset threshold, the association probability between the body bounding box and the hand bounding box that form the circumscribed box is decreased. Finally, the player hand corresponding to each player body is determined based on the association probability, namely the player body is paired with the player hand. As such, the player hand and body in the complex game place may be matched to monitor game currency placement or payment of the player or other processes in a game process more effectively.

[ 00190] In the embodiment of the disclosure, the degree of association between the body and hand in the circumscribed box is determined through the hand key point in the circumscribed box and the association probability between the body bounding box and the hand bounding box that form the circumscribed box, so that a facility for obtaining specific position information of a hand may be simplified, and meanwhile, the accuracy in association of the body bounding box and the hand bounding box in the circumscribed box may be improved.

[ 00191] Based on the abovementioned embodiments, the embodiments of the disclosure provide a body and hand association apparatus. FIG. 5 is a composition structure diagram of a body and hand association apparatus according to an embodiment of the disclosure. As shown in FIG. 5, the body and hand association apparatus 500 includes a first acquisition module 510, a first determination module 520, a second determination module 530, a third determination module 540, and a fourth determination module 550.

[ 00192] The first acquisition module 510 is configured to acquire an image to be detected of which an image content includes a body and a hand.

[ 00193] The first determination module 520 is configured to determine a body bounding box of the body and a hand bounding box of the hand in the image to be detected.

[ 00194] The second determination module 530 is configured to determine an association probability between the body bounding box and the hand bounding box.

[ 00195] The third determination module 540 is configured to determine a circumscribed box of the body bounding box and the hand bounding box. [ 00196] The fourth determination module 550 is configured to determine a degree of association between the body and hand in the circumscribed box based on a hand key point in the circumscribed box and the association probability.

[ 00197] In some embodiments, the second determination module 530 includes a feature determination sub-module 531 and an association probability determination sub-module 532.

[ 00198] The feature determination sub-module 531 is configured to determine a body feature of the body bounding box and a hand feature of the hand bounding box.

[ 00199] The association probability determination sub-module 532 is configured to determine the association probability between the body bounding box and the hand bounding box based on the body feature and the hand feature.

[ 00200] In some embodiments, the third determination module 540 includes a bounding box determination sub-module 541 and a circumscribed box determination submodule 542.

[ 00201] The bounding box determination sub-module 541 is configured to determine, as a target bounding box pair, a body bounding box and a hand bounding box between which an association probability is greater than a preset probability threshold.

[ 00202] The circumscribed box determination sub-module 542 is configured to determine a circumscribed box of the body bounding box and the hand bounding box in the target bounding box pair.

[ 00203] In some embodiments, the circumscribed box determination sub-module 542 includes a candidate box determination subunit 5421 and a circumscribed box determination subunit 5422.

[ 00204] The candidate box determination subunit 5421 is configured to determine a candidate box that encloses the target bounding box pair.

[ 00205] The circumscribed box determination subunit 5422 is configured to adjust a cover area of the candidate box based on a relative positional relationship between the body bounding box and the hand bounding box in the target bounding box pair to obtain the circumscribed box.

[ 00206] In some embodiments, the fourth determination module 550 includes a hand key point determination sub-module 551 and an association degree determination submodule 552.

[ 00207] The hand key point determination sub-module 551 is configured to determine at least two target hand key points in the circumscribed box, the at least two target hand key points including key points that belong to two different hands of a same person respectively.

[ 00208] The orrelation degree determination sub-module 552 is configured to determine, based on the at least two target hand key points, the degree of association between the body bounding box and the hand bounding box that form the circumscribed box.

[ 00209] In some embodiments, the association degree determination sub-module 552 includes a confidence determination subunit 5521, a distance determination subunit 5522, an association probability determination subunit 5523, and an association degree determination subunit 5524.

[ 00210] The confidence determination subunit 5521 is configured to determine a confidence that each target hand key point is a ground truth point.

[ 00211] The distance determination subunit 5522 is configured to, in a case that any confidence is greater than a preset threshold, determine a distance from the target hand key point corresponding to the confidence greater than the preset threshold to the hand bounding box that forms the circumscribed box.

[ 00212] The association probability determination subunit 5523 is configured to determine, based on the distance, a target association probability between the body bounding box and the hand bounding box that form the circumscribed box.

[ 00213] The association degree determination subunit 5524 is configured to determine, based on the target association probability, the degree of association between the body bounding box and the hand bounding box that form the circumscribed box.

[ 00214] In some embodiments, the association probability determination subunit 5523 is further configured to, in a case that the distance is less than or equal to a preset distance, determine the association probability between the body bounding box and the hand bounding box that form the circumscribed box to obtain an in-box association probability, and determine the in-box association probability as the target association probability.

[ 00215] In some embodiments, the association probability determination subunit 5523 is further configured to, in a case that the distance is greater than the preset distance, decrease the association probability between the body bounding box and the hand bounding box that form the circumscribed box to obtain a corrected association probability, and determine the corrected association probability as the target association probability.

[ 00216] In some embodiments, the association probability determination subunit 5523 is further configured to, in the case that the distance is greater than the preset distance, acquire a preset adjustment parameter configured to adjust the association probability, and lower the association probability between the body bounding box and the hand bounding box that form the circumscribed box using the preset adjustment parameter to obtain the corrected association probability.

[ 00217] In some embodiments, the association probability determination subunit 5523 is further configured to, in a case that the at least two target hand key points include no target hand key point of which the confidence is greater than the preset threshold, decrease the association probability between the body bounding box and the hand bounding box that form the circumscribed box to obtain the corrected association probability, and determine the corrected association probability as the target association probability.

[ 00218] In some embodiments, the target hand key point includes a wrist key point.

[ 00219] In some embodiments, in a case of determining multiple body bounding boxes and/or multiple hand bounding boxes in the image to be detected,

[ 00220] the bounding box determination subunit 541 is further configured to combine each detected body bounding box and each detected hand bounding box to obtain multiple bounding box pairs.

[ 00221] The circumscribed box determination sub-module 542 is further configured to determine a circumscribed box of the body bounding box and the hand bounding box in each bounding box pair.

[ 00222] The fourth determination module 550 is further configured to determine a degree of association between a body and hand in each circumscribed box to obtain a set of target degrees of association, determine a candidate degree of association greater than a preset association degree threshold in the set of target degrees of association, and based on a target circumscribed box corresponding to the candidate degree of association, determine a hand matching a body in the target circumscribed box. [ 00223] It is to be pointed out that descriptions about the above apparatus embodiment are similar to descriptions about the method embodiment, and beneficial effects similar to those of the method embodiment are achieved. Technical details undisclosed in the apparatus embodiment of the disclosure may be understood with reference to the descriptions about the method embodiment of the disclosure.

[ 00224] It is to be noted that, in the embodiments of the disclosure, when implemented in form of a software function module and sold or used as an independent product, the body and hand association method may also be stored in a computer-readable storage medium. Based on such an understanding, the technical solutions of the embodiments of the disclosure substantially or parts making contributions to the related art may be embodied in form of software product. The computer software product is stored in a storage medium, including a plurality of instructions configured to enable an electronic device (which may be a smart phone with a camera, a tablet computer, etc.) to execute all or part of the method in each embodiment of the disclosure. The storage medium includes various media capable of storing program codes such as a U disk, a mobile hard disk, a Read Only Memory (ROM), a magnetic disk, or an optical disk. Therefore, the embodiments of the disclosure are not limited to any specific hardware and software combination.

[ 00225] Based on the same technical concept, the embodiments of the disclosure provide a computer device, which is configured to implement a body and hand association method recorded in the method embodiment. FIG. 5 is a composition structure diagram of a computer device according to an embodiment of the disclosure. As shown in FIG. 6, the computer device 600 includes a processor 601, at least one communication bus, a communication interface 602, at least one external communication interface, and a memory 603. The communication interface 602 is configured to implement connections and communications between these components. The communication interface 602 may include a display screen, and the external communication interface may include a standard wired interface and wireless interface. The processor 601 is configured to execute a body and hand association program in the memory to implement the body and hand association method provided in the abovementioned embodiments.

[ 00226] Correspondingly, the embodiments of the disclosure provide a computer- readable storage medium, in which a computer program is stored, the computer program being executed by a processor to implement the steps in any body and hand association method as described in the abovementioned embodiment.

[ 00227] Correspondingly, there is also provided a chip in the embodiments of the disclosure. The chip includes a programmable logic circuit and/or a program instruction. The chip, when running, is configured to implement the steps in any body and hand association method as described in the abovementioned embodiments.

[ 00228] Correspondingly, there is also provided a computer program product in the embodiments of the disclosure. The computer program product, when executed by a processor of an electronic device, is configured to implement the steps in any body and hand association method as described in the abovementioned embodiments.

[ 00229] The above descriptions about the embodiments of the body and hand association apparatus, the computer device, and the storage medium are similar to the descriptions about the method embodiments, and technical descriptions and beneficial effects are similar to those of the corresponding method embodiments. Due to the space limitation, references can be made to the records in the method embodiments, and elaborations are omitted herein. Technical details undisclosed in the embodiments of the body and hand association apparatus, computer device, and storage medium in the embodiments of the disclosure may be understood with reference to the descriptions about the method embodiments of the disclosure.

[ 00230] It is to be understood that "one embodiment" and "an embodiment" mentioned in the whole specification mean that specific features, structures or characteristics related to the embodiment is included in at least one embodiment of the disclosure. Therefore, "in one embodiment" or "in an embodiment" mentioned throughout the specification does not always refer to the same embodiment. In addition, these specific features, structures or characteristics may be combined in one or more embodiments freely as appropriate. It is to be understood that, in each embodiment of the disclosure, a magnitude of a sequence number of each process does not mean an execution sequence and the execution sequence of each process should be determined by its function and an internal logic and should not form any limit to an implementation process of the embodiments of the disclosure. The sequence numbers of the embodiments of the disclosure are adopted not to represent superiority -inferiority of the embodiments but only for description. It is to be noted that terms "include" and "contain" or any other variant thereof is intended to cover nonexclusive inclusions herein, so that a process, method, object or device including a series of elements not only includes those elements but also includes other elements which are not clearly listed or further includes elements intrinsic to the process, the method, the object or the device. In a case of no more limitations, an element defined by the statement "including a/an " does not exclude existence of the same other elements in a process, method, object or device including the element.

[ 00231] In some embodiments provided by the disclosure, it is to be understood that the disclosed device and method may be implemented in another manner. The device embodiment described above is only schematic, and for example, division of the units is only logic function division, and other division manners may be adopted during practical implementation. For example, multiple units or components may be combined or integrated into another system, or some characteristics may be neglected or not executed. In addition, coupling or direct coupling or communication connection between each displayed or discussed component may be indirect coupling or communication connection, implemented through some interfaces, of the device or the units, and may be electrical and mechanical or adopt other forms.

[ 00232] The units described as separate parts may or may not be physically separated, and parts displayed as units may or may not be physical units, and namely may be located in the same place, or may also be distributed to multiple network units. Part of all of the units may be selected according to a practical requirement to achieve the purposes of the solutions of the embodiments.

[ 00233] In addition, each function unit in each embodiment of the disclosure may be integrated into a processing unit, each unit may also serve as an independent unit and two or more than two units may also be integrated into a unit. The integrated unit may be implemented in a hardware form and may also be implemented in form of hardware and software function unit. Those of ordinary skill in the art should know that all or part of the steps of the method embodiment may be implemented by related hardware instructed through a program, the program may be stored in a computer-readable storage medium, and the program is executed to execute the steps of the method embodiment. The storage medium includes: various media capable of storing program codes such as a mobile storage device, a ROM, a magnetic disk, or an optical disc.

[ 00234] Or, when being implemented in form of software function module and sold or used as an independent product, the integrated unit of the embodiments of the disclosure may also be stored in a computer-readable storage medium. Based on such an understanding, the technical solutions of the embodiments of the disclosure substantially or parts making contributions to the conventional art may be embodied in form of software product, and the computer software product is stored in a storage medium, including a plurality of instructions configured to enable a computer device (which may be a personal computer, a server, a network device or the like) to execute all or part of the method in each embodiment of the disclosure. The storage medium includes: various media capable of storing program codes such as a mobile hard disk, a ROM, a magnetic disk, or an optical disc. The above is only the specific implementation mode of the embodiments of the disclosure and not intended to limit the scope of protection of the embodiments of the disclosure. Any variations or replacements apparent to those skilled in the art within the technical scope disclosed by the embodiments of the disclosure shall fall within the scope of protection of the embodiments of the disclosure. Therefore, the scope of protection of the embodiments of the disclosure shall be subject to the scope of protection of the claims.