| JP2001101288 | DEVICE FOR PROVIDING MERCHANDISE INFORMATION |
| WO/1996/018925 | TRACKING SYSTEM FOR STEREOSCOPIC DISPLAY SYSTEMS |
| JP2003237490 | COMMUNICATION DEVICE FOR VEHICLE |
KONG, Xiaodong (())
| CLAIMS 1 . A method for controlling a contact-less system comprising an information device and a camera adapted to capture a live video of a user' hand, the method comprising: -a hand detecting step of detecting the user's hand from the live video captured by the camera to output location information of the user's hand; -a hand tracking step of generating a movement trace of the user's hand based on the location information; and -a trace analyzing step of analyzing the movement trace to detect a predefined trace pattern and generate a signal representing the trace pattern. 2. The method according to claim 1 , further comprising, a signal converting step of converting the signal into a command executable by a target system. 3. The method according to claim 1 or 2 , wherein the hand detecting step comprises : -reading a pre-stored template or model for the user's hand; -performing a match processing on each frame of the captured live video with the template or model; and -outputting the location information of the user's hand in each frame of the live video if there is a match between one or more frames of the live video and the template or model. 4. The method according to claim 1 or 2 , wherein the hand tracking step comprises: -acquiring the location information of the user's hand in each frame of the live video; and -generating the movement trace of the user's hand in a temporal order. 5. The method according to claim 1 or 2 , wherein the trace analyzing step comprises : -detecting the generated movement trace using a pre- stored predefined trace pattern; and -generating the signal representing the movement trace if the movement trace matches the predefined trace pattern. 6. The method according to claim 1 or 2 , further comprising, prior to the hand detecting step, a step of defining a hand gesture and a trace pattern depending on the user's requirement. 7. The method according to claim 6, wherein the step of defining a hand gesture and a trace pattern depending on the user's requirement comprises: -presenting, by the information device, a plurality of predefined trace patterns and a plurality of commands for the target system to the user; and - selecting, by the user, a desired trace pattern from the predefined trace patterns, and associating the selected trace pattern with one of the commands for the target system. 8. An information device provided with a camera and used in a contact-less system, the camera being adapted to capture a live video of a user's hand, the information device comprising: -a hand detecting unit adapted to detect the user's hand from the live video captured by the camera to output location information of the user's hand; -a tracking unit adapted to generate a movement trace of the user's hand based on the location information; and -a trace analyzing unit adapted to analyze the movement trace to detect a predefined trace pattern and generate a signal representing the trace pattern. 9. The information device according to claim 8, further comprising a signal converting unit adapted to convert the signal into a command executable by a target system. 10. The information device according to claim 8 or 9 , wherein the hand detecting unit reads a pre-stored template or model for the user's hand, performs a match processing on each frame of the captured live video with the template or model, and outputs the location information of the user's hand in each frame of the live video if there is a match between one or more frames of the live video and the template or model. 1 1 . The information device according to claim 8 or 9 , wherein the hand tracking unit acquires the location information of the user's hand in each frame of the live video and generates the movement trace of the user's hand in a temporal order. 12. The information device according to claim 8 or 9 , wherein the trace analyzing unit detects the generated movement trace by using a pre-stored predefined trace pattern, and generates the signal representing the movement trace if the movement trace matches the predefined trace pattern. 13. The information device according to claim 8 or 9 , further comprising a defining unit adapted to define a hand gesture and a trace pattern depending on the user's requirement. 14. The information device according to claim 13 , wherein the defining unit adapted to presents to the user a plurality of predefined trace patterns and a plurality of commands for the target system, and associates a desired trace pattern selected from the predefined trace patterns by the user with one of the commands for the target system. 15. The information device according to claim 8 or 9 , wherein the target system is integrated with the information device. 16. A contact-less system, comprising the information device according to claim 8 or 9. 17. A control program which causes a computer to perform a method for controlling a contact-less system comprising an information device and a camera adapted to capture a live video of a user' hand, the method comprising: -a hand detecting step of detecting the user's hand from the live video captured by the camera to output location information of the user's hand; -a hand tracking step of generating a movement trace of the user's hand based on the location information; and -a trace analyzing step of analyzing the movement trace to detect a predefined trace pattern and generate a signal representing the trace pattern. 18. A computer-readable storage medium storing a control program which causes a computer to perform a method for controlling a contact-less system comprising an information device and a camera adapted to capture a live video of a user' hand, the method comprising: -a hand detecting step of detecting the user's hand from the live video captured by the camera to output location information of the user's hand; -a hand tracking step of generating a movement trace of the user's hand based on the location information; and -a trace analyzing step of analyzing the movement trace to detect a predefined trace pattern and generate a signal representing the trace pattern. |
TITLE OF INVENTION :
INFORMATION DEVICE, CONTROL METHOD THEREOF AND
SYSTEM TECHNICAL FIELD
The invention relates to contact-less control, and more particularly, to an information device, a control method thereof and a system, which allow a user to control the information device in a contact-less manner by waving his or her hand.
BACKGROUND ART
It is well known that intelligent contact-less control is a prospective development trendy in the field of system control. Among various contact-less control methods, those based on visual information are very important since the visual information can provide a machine with a way of perceiving the world like human beings.
On the other hand, with the rapid development of manufacture technology, camera devices become cheaper and more powerful in performance . Now, a camera has become a standard accessory for many information devices. For example , cameras are mounted on various devices ranging from mobile phones to notebook computers and from automatic telling machine s (ATM s) to electronic bulletin boards . All of these developments have provided a solid foundation for applications based on visual information . However, currently in most cases , the camera has only some simple functions . For example , a camera on an ATM is only used for record visual information . Therefore, there is a need for developing more methods based on visual information, to extend the application scope of electronic devices equipped with cameras . Patent Document 1 (US5594469) disclo ses a hand gesture machine control system . In this system, a live video of a user is detected and captured by a camera. Then, the user's hand is detected in the live video . Also , an icon of the detected hand of the user can be displayed on a screen . As the user's hand moves, the icon is accordingly moved on the screen . In this way, parameters of a TV set, such as volume, contrast, channel or color, can be adjusted . In Patent Document 1 , the user's hand is detected by a camera provided on the TV set utilizing some relevant technique . The camera can remove the background from the video, recognize the hand gesture of the user and generate a command to be executed. Then , icons for various parameters , which vary depending on the hand gesture, can be displayed on a display. However, according to Patent Document 1 , the recognition of the user's hand gesture requires cutting away the background of the video. This causes some problems . For example , the user cannot be in front of the camera immediately after the system is started. Actually, at this moment no object should be in front of the camera, except a stationary background . Otherwise, the video captured by the camera will contain a large amount of noises, which leads to reduction in reliability of the subsequent recognition. Further, in Patent Document 1 , the environment illumination should remain unchanged during the operation of the camera. Once the environment illumination changes, a large amount of noises may also be introduced, which further reduces the reliability of the subsequent recognition.
Moreover, in Patent Document 1 , the TV set is controlled based on the movement direction of the hand. In this case, the camera provided on the TV set should be fixed during the whole process. Otherwise, the movement of the hand as captured by the camera may lead to erroneous operation of the TV set.
Therefore, there is a need for a hand-gesture-based control technique with higher reliability. SUMMARY OF INVENTION An object of the present invention is to provide an information device, a control method thereof and a system, which allow a user to control the information device in a contact-less manner by waving his or her hand.
According to a first aspect of the present invention, a method for controlling a contact-less system comprising an information device and a camera adapted to capture a live video of a user' hand, the method comprising: a hand detecting step of detecting the user's hand from the live video captured by the camera to output location information of the user's hand; a hand tracking step of generating a movement trace of the user's hand based on the location information; a trace analyzing step of analyzing the movement trace to detect a predefined trace pattern and generate a signal representing the trace pattern; and a signal converting step of converting the signal into a command executable by a target system.
According to a second aspect of the present invention, an information device provided with a camera and used in a contact-less system, the camera being adapted to capture a live video of a user's hand, the information device comprising: a hand detecting unit adapted to detect the user's hand from the live video captured by the camera to output location information of the user's hand; a tracking unit adapted to generate a movement trace of the user's hand based on the location information; a trace analyzing unit adapted to analyze the movement trace to detect a predefined trace pattern and generate a signal representing the trace pattern; and a signal converting unit adapted to convert the signal into a command executable by a target system.
According to a third aspect of the present invention, a contact-less system is provided, which comprises the above information device .
With the above device, system and method according to the present invention, it is possible to achieve a robust and reliable control of a target system in a contact-less manner.
BRIEF DESCRIPTION OF DRAWINGS
The above features and advantages of the present invention will be more apparent from the following detailed descriptions taken in conjunction with the figures, in which:
Fig. 1 shows a schematic diagram of a contact-less control system according to an embodiment of the present invention ;
Fig. 2 shows a schematic block diagram of a contact-less control system according to an embodiment of the present invention;
Fig. 3 is a flowchart illustrating a control method according to an embodiment of the present invention;
Fig. 4 is a flowchart illustrating a process of detecting a user's hand;
Fig. 5 is a flowchart illustrating a process of tracking a user's hand;
Fig. 6 is a flowchart illustrating a process of detecting a trace of a user's hand;
Fig. 7 is a flowchart illustrating a process of signal conversion;
Fig. 8 is a schematic diagram of dynamic hand gestures according to an embodiment of the present invention;
Fig. 9 is a schematic diagram of a command mapping table according to an embodiment of the present invention;
Fig. 10 shows a schematic process of controlling a TV set according to four hand gesture commands shown in Fig. 9 ;
Figs. 1 1A, 1 1 B and 1 1 C each show a schematic process of controlling a TV set with more hand gesture commands;
Fig. 12 shows a schematic diagram of a command mapping table in combination with handwriting recognition technique according to an embodiment of the present invention;
Fig. 13 shows a schematic process of controlling a TV set with the hand gesture commands shown in Fig. 12 ;
Figs . 14A, 14B, 14C and 14D each show a process of controlling a TV set with the commands shown in Fig. 12 ; and Fig. 15 shows a schematic diagram of a process in which a subj ect to be shot performs self-shooting using a camera according to an embodiment of the present invention.
DESCRIPTION OF EMBODIMENTS
The preferred embodiments of the present invention will be detailed in the following with reference to the figures. Throughout the figures, the same or similar elements, though shown in different figures, are represented by the same reference sign. For clarity and conciseness purpose, details for known functions and structures contained herein will be omitted, so as not to obscure the concept of the present invention.
Fig. 1 shows a schematic diagram of a contact-less control system according to an embodiment of the present invention. As shown in Fig. 1 , the contact-less control system according the embodiment of the present invention comprises an information device 100 (e .g. , a computer) , a camera 1 10 deployed exactly facing a user, and a presenting screen 120. In operation, the user's hand 130 is within the shooting range of the camera 1 10 so that a live video of the user's hand can be shot. The live video of the user's hand can be captured by the camera 1 10 and inputted to the information device 100. According to another embodiment of the present invention, the camera 1 10 can be integrated into the information device 100, such as a notebook computer with a camera or a digital camera.
The information device 100 can detect and track the user's hand. The position of a cursor displayed on the screen of the information device 100 is relocated in accordance with the movement trace of the user's hand. Then, the information device 100 compares the trace with a predefined pattern, so as to enable various controls such as left click or right click. In this way, the user 150 can control a target system connected to or within the information device by means of hand gesture.
Fig. 2 shows a schematic block diagram of the contact-less control system according to the embodiment of the present invention. As shown in Fig. 2 , when the user 150 waves his/ her hand 130, the camera 1 10 captures a live video and inputs it to the information device 100. The information device 100 generates, based on the live video, a control command corresponding to the hand-waving, and transmits it to the target system 140 for controlling the operation of the target system. As discussed above, the target system 140 can also form a part of the information device. one or more preset images of the user's hand can be stored in a memory unit 105 of the information device 100, along with at least one predefined trace pattern, such as "V, "Λ" and "O", corresponding to signals such as "left key click", "right ke click" and "double click", respectively.
As shown in Fig. 2 , a hand detecting unit 10 1 in the information device 100 receives the live video from the camera and performs a match processing on each frame of pictures in the live video with the preset image or model of the hand with. If there is one or more frames of pictures match the preset image or model, it is determined that there is an image of the user's hand in the live video.
The information device 100 is also provided with a hand tracking unit 102 which generates a graph of movement trace of the user's hand based on the detection result from the hand detecting unit 10 1. A trace analyzing unit 103 compares the trace pattern or patterns pre-stored in the memory unit 105 with the generated movement trace . Once a trace pattern is found to be matched with the movement trace, the trace analyzing unit 103 takes a command corresponding to the matched trace pattern as a command represented by the generated trace, and generates a signal representing the movement trace . Then, the signal is converted by a signal converting unit 104 into a command suitable for execution by the target system 140. As shown in Fig. 2 , the information device 100 is further provided with a defining unit 106 which enables the user 150 to customize images of his or her hand and the trace pattern(s) . When the user 150 wants to define images of his or her hand, he or she can use the camera 100 to capture images of his or her hand and store one or more of the captured images in the memory unit 105 as the template of his or her hand. Also, the user 150 can input specific trace pattern(s) , such as "m", to define his or her own trace pattern(s) .
In the following, the process of the control method and the operations of the respective units of the information device according to the embodiments of the present invention will be described with reference to the flowcharts. Fig. 3 is a flowchart illustrating the process of a control method according to an embodiment of the present invention.
At step S3 1 , the user 150 waves his or her hand in front of the camera 1 10, to control the cursor on the screen of the information device 100. At step S32 , the camera 1 10 in the information device 100 (e . g. , a computer) captures the live video of the user's hand and inputs it to the hand detecting unit 10 1 of the information device.
Next, at step S33 , the hand detecting unit 101 receives the live video from the camera and uses a preset image of the user's hand to perform a match processing on each frame of pictures in the live video. If there is a matched picture, it is determined that there is an image of the user's hand in the live video.
At step S34 , the hand tracking unit 102 generates a movement trace of the user's hand based on the detection result from the hand detecting unit 101 . The trace analyzing unit 103 compares the trace pattern(s) pre-stored in the memory unit 105 with the generated movement trace . Once a matched trace pattern is found, at step S35, the trace analyzing unit 103 takes the command corresponding to the matched trace pattern as a command represented by the movement trace, and generates a signal representing the movement trace . Finally, at step S36, the signal converting unit 104 converts the signal into a command suitable for execution by the target system 140.
Fig. 4 is a flowchart illustrating a process of detecting the user's hand. As shown in Fig.4, the hand detecting unit 101 detects, at step S4 1 , the captured live video based on a template of the user's hand stored in the memory unit 105 by using techniques such as template matching or texture matching, and determines whether there is an image of the user's hand at step S42. If there is not, the process goes to step S41 and continues detection. If there is an image of the user's hand, at step S43 , the hand detecting unit outputs the location of the hand in the image . The location can be used to locate the cursor on the screen and to generate a movement trace of the hand.
According an embodiment of the present invention, other algorithms, such as Adaboost algorithm or other object/ movement detection algorithms, can also be used to detect the user's hand in the live video.
As described above, prior to the step of hand detection, the image of the user's hand may be defined in advance , such that the defined hand image can be used in subsequent detection and tracking.
Fig. 5 is a flowchart illustrating a process of tracking a hand of a user. As shown in Fig. 5, information of the user's hand is obtained at step S5 1 before tracking. Then, at step S52 , the hand tracking unit 102 creates a tracker for the user's hand by using, for example, the mean shift tracking method, and initializes the tracker with information about the user's hand, such as color and texture. At step S53 , the hand tracking unit 102 starts to operate, and video frames are continuously updated. For each new frame, the tracker searches for an area most similar to it and matches information of the area with the information of the user's hand. According to an embodiment of the present invention, the hand tracking unit 102 can also employ other tracking method, such as the conditional probability density propagation method .
At step S54, once a stable tracking is achieved, the hand tracking unit 102 can obtain the location information and the area of the user's hand in each frame . In this way, based on the detected location information and area of the hand, the movement trace of the hand can be created in a temporal order.
Fig. 6 is a flowchart illustrating a process of detecting a trace of the user's hand. A trace pattern needs to be defined prior to detecting the trace of the user's hand. The trace pattern can be used to detect a trace of specific shape. For example, the user can create a unique trace pattern via the defining unit 106. Since the created pattern is unique to the user, the user can use his or her unique trace pattern to operate devices in different applications.
The trace analyzing unit 103 starts and acquires the detected trace from the hand tracking unit 102. Then, at step S62 , the trace pattern is analyzed based on all of the preset trace patterns stored in the memory unit 105. At step S63 , upon detection of a specific trace pattern, the trace analyzing unit 103 outputs a signal corresponding to the pattern. In this way, the user 150 can issue a signal to the target system by way of his or her hand gesture . According to an embodiment of the present invention, after obtaining the trace of the user's hand from the live video, robust features of the trace can be obtained by using a wavelet boundary descriptor or any other feature extraction algorithm. For example , these features are invariable with translation, scale and rotation. Further, these features can be used in recognizing the user's hand gesture. The trace analyzing unit 103 can use a Support Vector Machine (SVM) algorithm or any other classification algorithm for recognition. The recognition result is output to the signal converting unit 104.
Fig. 7 is a flowchart illustrating a process of signal conversion. The signal output from the trace analyzing unit 103 corresponds to the hand gesture of the user 150. Yet, the signal cannot be executed by the target system 140, since the target system 140 does not understand the signal. Thus, the signal converting unit 104 obtains the signal corresponding to the analyzed trace at step S71 , and converts the signal into a command in step S72. Afterwards, at step S73 , the signal converting unit 104 outputs the command. Fig. 8 is a schematic diagram of hand gestures according to an embodiment of the present invention. In this embodiment, 16 different traces are defined representing different commands, such as On, Off, Fast Forward, Back, Exit and the like, respectively. The user can select his or her preferred hand gestures to control an information device like a TV set. For example, the user may select the first hand gesture to represent an "On" command, the third hand gesture for a "Next" command, the fourth hand gesture for a "Previous" command and the second hand gesture for an "Off" command. Fig. 9 shows the correspondence between hand gestures and commands. With the user-defined hand gesture commands, the operation of the TV set can be controlled. Fig. 10 shows a process of controlling the operation of a
TV set with the hand gesture commands in Fig. 9. As shown in Fig. 10, when the TV set is in a standby state, the user can wave and make a hand gesture for issuing an "On" command to turn on the TV set. If the user wants to select the next channel, he or she can make a hand gesture for issuing a "Next" command to switch the TV channel to the next channel. If the user wants to select the previous channel, he or she can make a hand gesture for issuing a "Previous" command to switch the TV channel to the previous channel. At this time, if the user wants to turn off the TV set, he or she can make a hand gesture for issuing an "Off" command, and the TV set can enter the standby state upon receipt of this command.
The user can also add other hand gestures for controlling. For example, if the user wants to perform a more complicated operation, he or she can issue a "Call" command to call a system menu and then select a desired function. Figs. 1 1A, 1 I B and l l C each show an illustrative process of controlling a TV set with more hand gesture commands.
As shown in Fig. 1 1A, when a user wants to turn on a TV set, he or she can make a hand gesture for issuing an "On" command. In this case , the TV set is tuned to the first channel. When the user wants to switch the channel, he or she can make a hand gesture for issuing a "Next" command to switch the channel from the first channel to the second channel. After watching the second channel for a while, the user may want to switch back to the previous channel. In this case, the use can make a hand gesture for issuing a "Previous" command to switch the TV channel from the current channel to the previous channel, i. e . , the first channel. Afterwards, if the user wants to call a function menu, he or she can issue a "Call" command to invoke a system menu. The focus may be located on the first or any other default item in the system menu, e . g. , an "Exit" item as shown in Fig. 1 1A. In this case, as shown in Fig. 1 1 B , if the user wants to select the next item, he or she can make a hand gesture for issuing a "Next" command to switch the focus onto a "Volume" item. Then, the user can make a hand gesture for issuing an "On" command to adjust the volume. After the volume item is activated, a sliding block for volume is shown on the screen, which can be slid leftwards or rightwards for volume adjustment. At this time, the user can make a hand gesture for issuing a "Next" command to slide the sliding block rightwards, thereby increasing the volume. If the user wants to reduce the volume, he or she can make a hand gesture for issuing a "Previous" command to reduce the volume . After the volume adjustment, as shown in Fig. 1 1 C , the user can make a hand gesture for issuing an "Off" command to turn off the volume menu and return to the system function menu. Then, the user can issue an "Off" command again to turn off the system function menu and start normal display of programs on the first channel. Afterwards, if the user wants to turn off the TV set, he or she can make a hand gesture for issuing an "Off" command, and the TV set can enter the standby state. A process of operating a TV set by using a set of simple hand gestures has been described above . However, it will be appreciated that the TV set can be operated in a more complicated manner. For example, a signal issued by the user can be recognized by the above hand gesture recognition technique in combination with a handwriting recognition technique, and then converted into a corresponding command executable by the target system. Fig. 12 shows a schematic diagram of a command mapping table in combination with the handwriting recognition technique according to an embodiment of the present invention.
According to the embodiment of the present invention, a trace of the user's hand can be readily obtained by the camera during the process of tracking the movement of the user's hand. In other words, if the user writes a Chinese character or any other character in the air, the system can obtain the trace of the user's handwriting, and recognize it as a corresponding Chinese character or other character, which is in turn converted into a corresponding command. For example, the user may write in the air a Chinese character "JF ", " c "_h " , " T " or " ¾ " to issue the commands "On" , "Off", "Next", "Previous" or "Call" as mentioned above. Fig. 12 shows the correspondence between the characters and the commands. In this direct mode, the user may select a channel by directly writing in the air an Arabic numeral corresponding to the channel number.
As shown in Fig. 13 , the user can issue a command to the TV set for operation control by writing in the air the corresponding Chinese characters a Jf " , " T ", "-h " or
Figs. 14A, 14B , 14C and 14D each show a process of controlling a TV set with the commands in Fig. 12. As shown in Fig. 14C, a specific Chinese character is written in the air as a command for switching the TV set from the current operation mode to the direct mode . Accordingly, the focus in the function menu is switched to the "Direct Mode" item. In this case, the user can write in the air one of the Arabic numerals 1 , 2 , 3 , 4 , 5, 6, 7, 8, 9 , 10 or the like to select a corresponding channel. When the user wants to turn off the function menu, he or she can write in the air a Chinese character "^ " to exit from the direct mode, as shown in Fig. 14D .
Fig. 15 shows a schematic diagram of a process in which a subj ect to be shot performs self-shooting using a camera according to an embodiment of the present invention . As discussed above, any two of the camera 1 10, the information device 100 and the target device 140 or all of them can be integrated into a single device. For example, the present invention can be implemented in a digital camera to facilitate self-shooting for the user.
In the conventional self-shooting process, the user needs to first fix the camera to focus on a shooting view and set the camera to an automatic shooting mode . Then, the user leaves the camera, enters the shooting view, and waits to be shot. The camera automatically shoots when a time threshold for shooting is reached, so as to complete the self- shooting process.
However, there are some problems with the self-shooting approach. For example, it is difficult for the user to set a suitable time limit. If the time limit is set to be too short, the user will not be able to enter the shooting view before the camera automatically shoots. If the time limit is set to be too long, the user needs to keep a pose for a long time, resulting in an imperfect photo . All of these problems lead to a failed shooting. According to the embodiment of the present invention, the user can perform self-shooting by makes a hand gesture, without the need for setting a self-shooting time threshold. For example, the user can first define a hand gesture for self-shooting in the camera as a self-shooting command in the way as described above . During the process of self-shooting, the user first sets the camera to the self-shooting mode, fixes the camera to focus on the shooting view, and then enters the shooting view. When the subjects to be shot, including the user, are ready, the user makes the defined hand gesture to issue a shooting command, and then immediately keep a desired pose . Upon detection of the self-shooting command in the way as described above, the camera carries out a self-shooting action immediately or with a delay of a few seconds, e.g. , two seconds. In this way, enough time can be reserved for the user to keep a pose while avoiding self-shooting failure due to a long waiting time .
According to the embodiments of the present invention, the hand gesture detection technique is replaced with a dynamic hand tracking technique, thereby achieving a more reliable system control. With the methods according to the present invention, the user can issue different commands to the target system by using different trace patterns to enable different operations. Since the trace patterns are unique both temporally and spatially, it is possible to ensure a reliable and accurate system control.
Moreover, according to the embodiments of the present invention, an image can be captured with a common PC camera. There is no need for addition of another camera device or video card.
As illustrated above, the device and method according to the present invention can be applied to an information apparatus supporting a camera function, such as desktop PC , laptop PC , mobile phone , PDA, electronic white board, remote control device, monitor device and the like.
The foregoing only describes the exemplary embodiments for implementing the present invention. It will be appreciated by those skilled in the art that any modifications and partial alternations can be made without departing from the scope of the present invention. These modifications and alternations should be within the scope of the present invention which is only limited by the appended claims.