BACKGROUND ART Recently, what is called, a DDR game apparatus has been all the fashion.

It has a music play device and a floor body sensing the foot action of a game player. In this DDR game apparatus, the foot action in which the game player will follows, is informed to the game player by either a monitor or a foot action direction of the floor body together with music played by the music play device, and the game player follows the foot action in accordance with timing.

As an apparatus further advanced than the DDR game apparatus, another game apparatus in which a sensor for sensing hand stretching action as well as the foot action of the game player is installed at a certain position thereof has been introduced. It can check the motions of both foot and hand actions using the sensor.

However, there are problems that it is impossible to display natural motion of the original actor and to evaluate overall motion of the game player following the motion, since the above described conventional game apparatuses can only sense the foot action or the foot action and hand stretching actions at a designated position.

DISCLOSURE OF INVENTION It is therefore one object of the present invention to provide a method of playing a motion data in which a game player can easily follow the motion of an original actor in a motion game apparatus displaying the motion of the original actor, for solving the above problems.

Furthermore, it is the other object of the present invention to provide a method of evaluating a motion data capable of real-time and rapidly evaluating the motion of a game player following the motion of an original actor in a motion game apparatus displaying the motion of the original actor.

The one object of the present invention can be achieved by a method of displaying a motion data in a motion game apparatus having information of a basic frame displaying main motion of an original actor, and playing the motion data of the original actor consisting of a plurality of frames on a display device, comprising: a first step of setting play position points Xf, Yf and Zf on the display device for playing continuous action of the original actor, and initial position points Xi, Yi and Zi on the display device for displaying beforehand the basic frame which will be played after a predetermined time At ; a second step of displaying the continuous action of the original actor at the play position points Xf, Yf and Zf based on present time t, and simultaneously displaying the basic frame at the initial position points Xi, Yi and Zi after drawing out the basic frame which will be displayed at t + At ; and a third step of displaying with gradually moving the basic frame displayed at the initial position points Xi, Yi and Zi to the play position points Xf, Yf and Zf as time goes by.

Furthermore, the one object of the present invention can be achieved by a method of displaying a motion data in a motion game apparatus having information of a basic frame displaying main motion of an original actor, consisting of a plurality of frames, having play position points Xf, Yf and Zf on a display device for playing continuous action of the original actor and initial

position points Xi, Yi and Zi on the display device for displaying beforehand the basic frame which will be played after a predetermined time At, and displaying the motion data of the original actor on the display device, comprising; a first step of displaying continuously the continuous action of the original actor at the play position points Xf, Yf and Zf based on a present time t, and simultaneously displaying the basic frame at the initial position points Xi, Yi and Zi after drawing out the basic frame which will be displayed at t+At ; and a second step of displaying with gradually moving the basic frame displayed at the initial position points Xi, Yi and Zi to the play position points Xf, Yf and Zf as time goes by.

Moreover, the other object of the present invention can be achieved by a method of evaluating the motion of a game player following the motion data of an original actor displayed on a display device in a motion game apparatus having the information of a basic frame displaying main motion of the original actor, consisting of a plurality of frames, and setting play position points Xf, Yf and Zf on the display device for playing continuous action of the original actor and initial position points Xi, Yi and Zi on the display device for displaying beforehand the basic frame which will be played after a predetermined time At, comprising; a first step of storing the three-dimensional motion data of the original actor retargeted after converting the three-dimensional motion data of the original actor by reflecting the body size of the game player; a second step of displaying continuously the continuous action of the original actor at the play position points Xf, Yf and Zf based on present time t using the retargeted three-dimensional motion data of the original actor and simultaneously displaying the basic frame at the initial position points Xi, Yi and Zi after drawing out the basic frame which will be displayed at t + At ; a third step of displaying with gradually moving the basic frame displayed at the initial position points Xi, Yi and Zi to the play position points Xf, Yf and Zf as time goes by; a fourth step of extracting and storing the three-dimensional motion data of the game player learning the same

motion as the original actor with following the motion of the original actor displayed; a fifth step of calculating the coordinate value of a reference sensor for the original actor corresponding to a sensor position attached to the game player from the basic frame displayed at the play position points Xf, Yf and Zf of the third step; and a sixth step of comparing the three-dimensional motion data of the game player with the coordinate value of the reference sensor.

The nature, utility, and further features of the present invention will be clearly apparent from the following detailed description with respect to preferred embodiments of the invention when read in conjunction with the accompanying drawings briefly described below.

BRIEF DESCRIPTION OF THE DRAWINGS Further objects and other advantages of the present invention will be apparent from the following description in conjunction with the attached drawings, in which: FIG. 1 is a configuration diagram of a motion game apparatus of the present invention.

FIG. 2 is a block diagram of a motion game apparatus applied to the present invention.

FIG. 3 is a flow chart showing retargeting processing sequence.

FIG. 4 is a frame structure diagram for describing one example of an image frame configuration of a three-dimensional original actor according to the present invention.

FIG. 5 and FIG. 6 are description drawings for describing a method of displaying preparation action of an original actor.

FIG. 7 is a flow chart showing overall action displaying and evaluating a motion using a motion frame of FIG. 4.

FIG. 8 is a flow chart describing a method of evaluating a motion of a game player.

FIG. 9a to FIG. 9d are a flow chart showing performance sequence of the present invention in detail.

DETAILED DESCRIPTION OF THE INVENTION Preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

Hereinafter, the present invention will be explained in detail with reference to the accompanying drawings.

FIG. 1 is a configuration diagram of a motion game apparatus of the present invention. The motion game apparatus includes a plurality of cameras 100, a display device 200 displaying a motion of an original actor, an input section receiving input from a game player, and a sound device. The game player attaches a plurality of optical sensors (not shown) to his body and takes a motion in a certain region range capable of be sensed by the camera. The camera 100 monitors the motion of the game player using the optical sensors attached to the body of the game player.

Preferably, the optical sensors are attached to all articulation parts of the game player if possible. More preferably, the number of the optical sensors is appropriately selected in view of processing rate because the number of image processing operations increase as the number of the optical sensors increases.

Furthermore, while it is possible to obtain precise data as the number of the camera increase, it is preferable to appropriately select the number of the cameras in view of the same problem as the number of the sensors above described.

FIG. 2 is a block diagram of a motion game apparatus applied to the present invention. The motion game apparatus of the present invention includes an input section receiving an input from a game player, a control section, a

sensor/camera, an image output device, a music D/B, a character/stage DB, a sound output device, and an image memory. The control section has a motion selection section outputting a motion selection signal in accordance with an output of the input section, a motion data storage section storing a plurality of motion data of an original actor, a retargeting processing section retargeting the motion of the original actor, a retargeted original actor motion data storage section storing a retargeted motion data, a game player motion data storage section storing the motion data of the game player, a motion evaluation section evaluating after comparing the retargeted motion data of the original actor with the motion data of the game player, a motion capture section capturing the motion of the game player, and a resultant image output section displaying the output of the motion evaluation section on the image output device.

The operation flow of a motion game apparatus in FIG. 2 is will be now explained.

Firstly, the game player selects one of the plurality of motion data using the input section. The motion select section selects the one of the plurality of motion data stored to the motion data storage section and then inputs the selected data to the retargeting processing section.

The retargeting processing section receives a body size from the game player or senses the body size by image processing using the camera, extracts the body size difference between the game player and the original actor, carries out retargeting process for the motion data of the original actor, and stores the retargeted motion data of the original actor in the motion data storage section.

Specific description with respect to the retargeting will be described in FIG. 3 later.

The retargeted motion of the original actor is provided to the game player by the image processing section, the image memory, and the image output device. At this time, the basic frame for main motion which will be provided, is provided to the game player by a frame storage section and a sliding control section so that the

game player can easily follow the motion of the original actor. The motion of the game player is stored in the game player motion data storage section real-timely.

The motion evaluation section then evaluates the motion of the game player after comparing the motion of the game player with the retargeted motion data of the original actor. Finally, the result is provided to the game player by the image output device.

To solve problems which may be occurred due to the body size difference between the original actor and the game player prior to analyzing the motion, the retargeting must be performed. That is to say, the retargeting is a processing to remove the body size difference.

The processing sequence of the retargeting will be explained with reference to FIG. 3.

Firstly, two data with respect to body sizes of the original actor and game player are received by input from a user or image capture of the camera using an image processing. The two data are analyzed, and the motion data of the original actor is converted so as to be appropriate to the body size of the game player. In general, the three-dimensional motion data of the original actor is magnified or reduced based on the specific part (mostly, waist) of the three-dimensional motion data of game player and then the center of magnified or reduced the three-dimensional motion data of the original actor is moved. After moving the center, finally, the retargeted motion data of the original actor may be obtained by performing conversion to satisfy a restriction condition. The reason why the conversion is required to satisfy the restriction condition is that foot is floated in the air, if a three-dimensional motion data of a basketball player with height of 2m is reduced based on his waist with respect to an elementary pupil with height of lm. Therefore, the restriction condition that"the foot is necessarily touched to the ground"is required, and the three-dimensional motion data of the basketball player must be converted considering this restrictive condition.

FIG. 4 is a frame structure diagram for describing one example of an image frame configuration of a three-dimensional original actor according to the present invention.

In FIG. 4, each of the boxes designates a motion frame stored to unit time intervals and each of the numbers on the boxes shows a frame number. It should be understood that the image frame of an original actor consists of a start frame, an end frame and motion frames. The motion frames are divided into basic motion frames and common motion frames. A special motion (namely, main motion of dance actions) of these motion frames is defined as the basic motion frame. The information with respect to the basic motion frame may be marked using a management program marking"a specific pose"by an expert (content maker) after a point of time creating a motion data or storing the motion data. The information with respect to the basic motion frame may be included in a portion of the motion frame displaying the motion of the original actor or be used as an additional file.

The motion frame is generally stored in the form of file to display a motion image such as for example, a BVH file. The format of the BVH file firstly describes about a triple structure consisting of a several sensors for designating each of the parts of the body, and then displays formations on an angle and a coordinate value versus time of the sensors corresponding to each of the parts of the body. The coordinate versus time may be understood by this BVH file.

FIG. 5 and FIG. 6 are description drawings for describing a method of displaying preparation action of an original actor. In FIG. 5, (a) shows an original actor motion frame displaying the motion of the original actor, (b) is a basic motion frame displaying an information on the basic motion of the motion frame of the original actor, and (c) shows a game player motion frame storing the motion of a game player following the motion of the original actor, respectively.

As shown in FIG. 5 (b), it will be understood that 5th, 10th, 17th, 19th, 21st, 25th,

27th, 28th, 31st, 37th, 38th, 40th, 42nd and 48th frames are designated as the basic frames. As present time t is the point of time when the game player must follow the motion of the original actor, the real-time motion of the original actor is provided. Furthermore, At is to display time interval for detecting the basic motion frame. According to FIG. 5, At shows the state that 6th frame is ruled of the time displayed. Furthermore, according to FIG. 5, the 16th motion frame of the original actor is provided to the game player at the present time t, and the basic motion frames (17th, 19th and 21st frames) between t and t+At are simultaneously displayed, for showing beforehand to the game player.

For example, FIG. 6 is description drawing for describing a method of displaying such points with a display device. As shown in FIG. 6 (a), continuous motions (the 16th frame in FIG. 5) of the original actor in which a present game player must follow, are displayed on one region 61 of a screen, and main motions (17th, l9th and 21st frames in FIG. 5) in which the game player will follow at next stage are sequentially and real-timely displayed according to time on the other regions 62,63 and 64 of the screen. By using this display method, the game player identifies beforehand next main motions, so that he can easily follow the motions of the original actor.

As shown in FIG. 6 (a), when setting a coordinate on the screen based on a coordinate axis, in case that the eye height He of the game player is longer than distance Dy between the foot position of the game player and the center position on the screen of the display device, a coordinate value Yi on Y axis starting a guide action and a coordinate value Yf on Y axis of the screen displaying real-timely continuous actions must satisfy following equation (1).

Yi > Yj.................................... equation (1) This is to further improve the property of a three-dimensional image by

displaying the guide action in view of the eye height of the game player.

Similarly, in case that the eye height He of the game player is shorter than distance Dy between the foot position of the game player and the center position on the screen of the display device, the coordinate value Yi on Y axis starting the guide action and the coordinate value Yf on Y axis of the screen displaying real-timely continuous actions must satisfy following equation (2).

Yi < yj.................................... equation (2) At this time, the direction of a vector consisting of starting points Xi, Yi and Zi and end points Xf, Yf and Zf can be easily determined by analyzing motion data such as BVH of the original actor including the position values of the sensor attached on the articulation parts of the original actor. Therefore, when displaying the guide action in view of the direction of the vector, it is possible to display the guide action more actually. Furthermore, it is preferable to locate position points Xf, Yf and Zf on the screen displaying the continuous actions of the original actor on the center of the screen, for providing the game player with facility. Moreover, it is preferable to appropriately locate start points Xi, Yi and Zi in view of the number of preparation action frame which will be displayed and horizontal size of the screen. The eye height He of the game player can be easily calculated using statistical value from the height of the game player to his eye position. Here, the height of the game player is input by the game player or operated by a camera.

FIG. 7 is a flow chart showing overall action displaying and evaluating the motion using the motion frame of FIG. 4. A motion data, a music DB and a character DB are selected from an input section by the game player (step S71).

Next, the body size of an original actor is compared with that of the game player using the body size inputted by the game player or obtained by analyzing the

image of the camera (step S72). A three-dimensional image coordinate value data of the original actor is converted using this comparison value (step S73). At this time, since the number and position of sensors attached on the game player are different from the number and position of sensors attached on the original actor for capturing the motions of the original actor, it is necessary to primarily agree with targets which will be compared for the purpose of evaluating the targets appropriately. Accordingly, the position data of the reference sensor is calculated from the motion data of the original actor based on the number and position of the sensors attached on the game player (step S74).

Thereafter, the three-dimensional motion data of the original actor is real-timely displayed, and the guide action is simultaneously displayed (step 75).

While the three-dimensional motion of the original actor may be displayed prior to retargeting the motion of the original actor, it is preferable to display the three-dimensional motion of the original actor after retargeting. In case of displaying the three-dimensional motion of the original actor prior to retargeting, while it is possible to substantially display the motion of the original actor, it is difficult to exactly evaluate the motions of the game player and original actor. In case of displaying the three-dimensional motion of the retargeted original actor, it is possible to easily compare the motion of the game player with the motion of the original actor. Namely, in case of displaying the motion of the game player by overlapping the motion of the original actor with the motion of the game player, there is advantage that the game player may easily identify how different from the motion of the original actor by using the retargeted motion.

The motion of the game player following the motion of the original actor displayed on the screen is stored real-timely by the frame (step S76). At this time, whether or not the three-dimensional motion frame of the original actor displayed at time t, is the basic motion frame, is determined (step 77). In case of the basic motion frame, the motion frame of the game player identified real-timely is

compared with the position data of the reference sensor calculated at the step 74 and evaluated (step S78). The steps S76 and S77 may be performed regardless of sequence. The step S74 may be also performed after step S77. Namely, whether or not the three-dimensional motion frame of the original actor is the basic motion frame, is determined, and with calculating the position data of the reference sensor for the basic motion frame, the calculated resultant value can be compared with the sensor position obtained from the motion of the game player and evaluated. When either the step S76 or the step S74 is performed after the step S77, only the motion frame of the game player following the basic motion frame of the original actor is stored, so that only the position data of the reference sensor will be operated.

Thereafter, whether or not the frame is the last frame, is determined (step S79). In case of the last frame, the step S75 is again started, while in case of not being the last frame, the final result is displayed using the resultant value of the comparison and evaluation (step S80).

Since it must be performed real-timely within a short time to compare and evaluate whether or not the motion of the game player is agreed with the basic motion frame of the original actor, it is performed by procedure proposed in FIG. 8 for the purpose of reducing time required for an image evaluation procedure.

1. The step of operating the position value of the reference sensor (step S81) As above described, to appropriately evaluate targets in the number and position of sensors attached on the original actor for capturing his the motion and attached on the game player, it is necessary to primarily agree with the comparison targets. Accordingly, the three-dimensional position value of the reference sensor is operated from the reference motion data of the original actor based on the number and position of the sensors attached on the game player. This is defined as the position value of the reference sensor. It is assumed that the position value

of the reference sensor for a sensor 1 attached on the game player is (XI, Y1, Z1), for facilitating description.

2. The step of calculating two-dimensional coordinate values corresponding to each of the cameras (step S82) As above described, a plurality of cameras are used for identifying the three-dimensional positions of the sensors attached on the game player. The three-dimensional image is obtained by synthesizing the two-dimensional image obtained from the plurality of the cameras.

By inversely performing procedure the same as method obtaining the three-dimensional image from the two-dimensional image, the two-dimensional coordinate value of corresponding to sensor position included in each of the cameras for embodying corresponding to the three-dimension image of the original actor, can be calculated. A two-dimensional coordinate value (X11, Y11) displayed on a first camera, a two-dimensional coordinate value (X21, Y21) displayed on a second camera,..., a two-dimensional coordinate value (Xnl, Ynl) displayed on a nth camera, can be respectively calculated from the reference position value (Xl, Y1, Z1) of the step S81 by this image processing method.

3. The step of calculating a target region of each of cameras (step S83) A certain region is set at a exact position where a sensor is discovered by each of the cameras. For example, when a game player exactly follows the motion of an original actor, the exact position value of the sensor 1 discovered by the camera 1 must be (X11, Y11). However, it is impossible that the game player exactly follows the motion without a selected error. Accordingly, when setting a certain region having for example, margin of 5 based on (X11, Y11), a square region consisting of (X11-5, Y11-5), (X11 + 5, Y11-5), (X11 + 5, Y11 + 5), and (X11-5, Y11 + 5) is selected. The selected region is referred as a target region for facility.

4. The step of checking whether or not the sensor is detected in the target

region (step S84) Whether or not the image coordinate of the sensor attached on the game player is introduced in the target region set at step S83 among the image data of each of the cameras is checked. As a conventional image process method, in case using the method that overall region of image data outputted from each of the cameras is checked, the position value of the sensor attached on the game. player is identified, and compared with the sensor position value of the original actor, long time is required for the image processing. Accordingly, the target region where the sensor will be discovered is set beforehand. Furthermore, by only checking not the overall region but a certain target region, image processing time can be considerably reduced. When checking the certain region in this manner, it can be easily evaluated that in case the sensor is discovered in the target region of all the cameras, the motion of the game player is the same as motion of the original actor, while in case the sensor is not discovered, the motion of the game player is different from the motion of the original actor. Furthermore, the three-dimensional position information can be obtained by combining the two-dimensional data detected in this manner. Namely, a method that the three-dimensional information which will be finally used, is calculated by the two-dimensional detection information obtained from at least two cameras based on one sensor, and the three-dimensional data of the original actor is then compared with the calculated information, can be used. Since an algorithm in which a three-dimensional information is obtained from a plurality of two-dimensional information, follows a conventional theory of a computer vision field, the description of the above algorithm is omitted here.

FIG. 9 is a flow chart showing performance sequence of the present invention in detail. All the body action of an user can be pursued using music in accordance with selection of the user and high performance motion capture system

(steps S101 and S102). A character and stage which can be applied to the case viewing this apart from the image of the original actor are then selected (steps S103 and S 104). Next, the motion data of the original actor is read from his the motion capture DB (step S105), and a character image is overlaid on the motion data of the original actor (step S106). Since the size of the image file increases when using the image of the original actor according to that, the amount of data processing can be reduced by overlaying the character image stored at the motion coordinate of the original actor beforehand. Next, an operation matching the motion data with the music data is performed (step S107). A display device is used for the motion data of the original actor and a music is outputted by a sound device (step S108). The user takes motion according to the output data (step S 109). Thereafter, all the body motions are captured by a plurality of cameras, a mark is identified from the image, and the motion is pursued by the frame (steps S110 and Sill). The two-dimensional coordinate value is extracted using the image data obtained by this method, and the three-dimensional motion coordinate value is obtained from the extracted data (steps S 113 and S 114) Next, using the coordinate data of the original actor and the three-dimensional coordinate value of the learner motion-captured, the body size of the original actor is compared with that of the learner (steps S115 and S116).

When the two body sizes are different from each other in real comparative resultant, retargeting is performed (steps S 117 and S 118). The retargeting can be performed during the motion-capture of the learner in accordance with flow as shown in FIG. 9, or prior to carrying out the motion-capture. After retargeting, in case of the basic frame, whether or not the data of the original are agreed with the data of the learner is checked by comparing with the two data (steps S119 and S120). When the two data are agreed with each other, a score is given, and the total time agreed is counted, thereby displaying advertisement objects obtained using action agreement message or advertisement object production DB (steps

S122 to S124). A motion capture data is stored in a user motion storage section, and then whether or not the frame is a last frame is checked (steps S125 and S126).

Thereafter, while in case of not being the last frame, next motion is captured, in case of the last frame, the game is over.

When the motion data of the original actor is not agreed with the motion data of the learner, after checking whether or not a function which the game is over in accordance with disagreement time is set (step S127), in case that the two data are not agreed with each other more than certain times, the game is over intentionally.

As above described, according to the present invention, the motion of a game player following the motion of a dancer can be real-timely evaluated by a motion game apparatus.

Furthermore, a method playing a motion data is proposed so that the game player can easily follow the motion of the original actor in the motion game apparatus displaying the motion of the original actor, thereby improving facility of user. Moreover, the motion of the game player following the motion of the original actor is real-timely and rapidly evaluated, thereby enabling the motion game apparatus to be utilized.

The invention may be embodied in other specific forms without departing from the spirit or essential characteristics thereof. The present embodiments are therefore to be considered in all respects as illustrative and not restrictive, the scope of the invention being indicated by the appended claims rather than by the foregoing description and all changes which come within the meaning the range of equivalency of the claims are therefore intended to be embraced therein.