TECHNICAL FIELD

The present invention relates to a method and system realized by a processor unit for providing making selection between said menu items of a user menu displayed in a display unit and comprising pluralities of menu items.

PRIOR ART

User interfaces are controlled by means of pluralities of methods and process is realized. User interface control can be provided according to the gestures of the user by realizing image processing and the user interface can also be controlled in accordance with the inputs received from physical input units like mouse, keyboard, etc.

In document US2009027337A1 , a method is disclosed, which permits detection of the face, body, arm and hand of the user and surfing in the user interface and making selection according to the movement of the arm and hand of the user. In this solution, the user has to make difficult physical movements besides the movements which will be realized by the user in the normal flow of life in order for the user to be able to control the interface. It is difficult for the users, who have limited physical movement capacity, to realize these movements.

As a result, because of the abovementioned problems, an improvement is required in the related technical field.

BRIEF DESCRIPTION OF THE INVENTION

The present invention relates to a system and method for user interface control, for eliminating the abovementioned disadvantages and for bringing new advantages to the related technical field.

An object of the present invention is to provide a system and method which provide controlling of the user interface by means of gesture without needing difficult physical movements by the user. Another object of the present invention is to provide a system and method for user interface control where less system source is used.

Another object of the present invention is to provide a system and method for user interface control where the proportion of misperception of the user movements is reduced.

In order to realize the abovementioned objects and the objects which are to be deducted from the detailed description below, the present invention is a method realized by a processor unit for providing making selection between said menu items of a user menu displayed in a display unit and comprising pluralities of menu items. Accordingly, the improvement is that the subject matter method comprises the following steps:

- receiving at least one medium image from an image sensor which captures the images of the medium where the user exists,

- detecting at least one face in the received medium image,

- detecting a thumb upward pattern in a medium image where the face is detected,

- determining finger lines which are the regions where the fingers contact each other in the medium image where the thumb upward pattern is detected,

- determining an angle between at least one of the finger lines and a predetermined reference axis,

- marking one of the menu items in accordance with the determined angle,

- searching a clicking movement in the medium image,

- providing selection of the marked menu item in case the clicking movement is detected. Thus, the user moves in a reduced manner and controls the user interface.

In a possible embodiment of the present invention, the step of “searching a clicking movement in the medium image” comprises the sub-steps of:

- determining a first region having predetermined dimensions and where the thumb upward pattern detected in the medium image exists,

- calculating the number of pixels which are in predetermined skin colors and which exist in the determined first region,

- selecting the marked menu item when it is detected that the number of pixels, which are in skin color, of the final medium image deviates at a predetermined amount from the pixel number of a prior medium image or when it is detected that pluralities of medium images deviate at a predetermined amount from the pixel number of a prior medium image.

In another possible embodiment of the present invention, the step of “searching a clicking movement in the medium image” comprises the sub-steps of: - determining a first region having predetermined dimensions and where the thumb upward pattern detected in the medium image exists,

- rotating the determined first region such that an upper part, which is the part of the thumb upward pattern comprising the thumb, corresponds to a predetermined reference point,

- counting the pixels which are compliant to the skin color at a predetermined range at the upper part,

- determining that clicking movement is realized in case it is detected that the number of skin color pixels existing at the upper part has decreased at predetermined proportion when compared with the number of skin color pixels existing at the upper part of the prior medium images.

In another possible embodiment of the present invention, after the step of “detecting a thumb upward pattern in a medium image where the face is detected”, the following sub-steps are provided:

- realizing detection of the thumb upward pattern instead of face detection process in each following medium image,

- passing to the step of “taking at least one medium image from an image sensor which captures the images of the medium where the user exists” in case the thumb upward pattern cannot be detected.

Thus, additional system source is not used for detecting face in each medium image and only thumb upward movement is detected and monitored.

In another possible embodiment of the present invention, said predetermined skin color is selected as the color of the detected face and as the colors which have tones close to this color.

In another possible embodiment of the present invention, the step of “determining a first region having predetermined dimensions and where the thumb upward pattern detected in the medium image exists” comprises the sub-steps of:

- determining the dimensions of the detected face,

- determining the dimension of the detected thumb upward pattern,

- passing to the step of taking medium image if it is detected that the proportions of the face dimension and the thumb upward dimension to each other deviate from the predetermined threshold values.

Thus, erroneous detection of the objects and visuals, existing in the medium, as thumb upward movement is prevented. In another possible embodiment of the present invention, the step of “determining a first region having predetermined dimensions and where the thumb upward pattern detected in the medium image exists” comprises the sub-steps of:

- inquiring whether the proportion of the number of pixels, which are in skin color and which are within predetermined ranges, of the detected thumb upward pattern to the number of pixels of thumb upward pattern exceeds a predetermined value,

- passing to the step of “taking at least one medium image from an image sensor which captures the images of the medium where the user exists” if the calculated proportion does not exceed the predetermined value.

The present invention is moreover a user interface control system comprising a processor unit for providing making selection between said menu items of a user menu displayed in a display unit and comprising pluralities of menu items. Accordingly, the improvement is that said processor unit is configured to realize the following steps:

- receiving at least one medium image from an image sensor which captures the images of the medium where the user exists,

- detecting at least one face in the received medium image,

- detecting a thumb upward pattern in a medium image where the face is detected,

- determining finger lines which are the regions where the fingers contact each other in the medium image where the thumb upward pattern is detected,

- determining an angle between at least one of the finger lines and a predetermined reference axis,

- marking one of the menu items in accordance with the determined angle,

- searching a clicking movement in the medium image,

- providing selection of the marked menu item in case the clicking movement is detected.

In another possible embodiment of the present invention, the processor unit is configured to realize the following sub-steps in the step of “searching a clicking movement in the medium image”:

- determining a first region having predetermined dimensions and where the thumb upward pattern detected in the medium image exists,

- calculating the number of pixels which are in predetermined skin colors and which exist in the determined first region,

- selecting the marked menu item when it is detected that the number of pixels, which are in skin color, of the final medium image deviates at a predetermined amount from the pixel number of a prior medium image or when it is detected that pluralities of medium images deviate at a predetermined amount from the pixel number of a prior medium image. In another possible embodiment of the present invention, the processor unit is configured to realize the following sub-steps in the step of “searching a clicking movement in the medium image”:

- determining a first region having predetermined dimensions and where the thumb upward pattern detected in the medium image exists,

- rotating the determined first region such that an upper part, which is the part of the thumb upward pattern comprising the thumb, corresponds to a predetermined reference point,

- counting the pixels which are compliant to the skin color at a predetermined range at the upper part,

- determining that clicking movement is realized in case it is detected that the number of skin color pixels existing at the upper part has decreased at predetermined proportion when compared with the number of skin color pixels existing at the upper part of the prior medium images.

In another possible embodiment of the present invention, the processor unit is configured to realize the following sub-steps after the step of “detecting a thumb upward pattern in a medium image where the face is detected”:

- realizing detection of the thumb upward pattern instead of face detection process in each following medium image,

- passing to the step of “taking at least one medium image from an image sensor which captures the images of the medium where the user exists” in case the thumb upward pattern cannot be detected.

In another possible embodiment of the present invention, said predetermined skin color is selected as the color of the detected face and as the colors which have tones close to this color.

In another possible embodiment of the present invention, the processor unit is configured to realize the following sub-steps after the step of “determining a first region having predetermined dimensions and where the thumb upward pattern detected in the medium image exists”:

- determining the dimensions of the detected face,

- determining the dimension of the detected thumb upward pattern,

- passing to the step of taking medium image if it is detected that the proportions of the face dimension and the thumb upward dimension to each other deviate from the predetermined threshold values. In another possible embodiment of the present invention, the processor unit is configured to realize the following sub-steps in the step of “determining a first region having predetermined dimensions and where the thumb upward pattern detected in the medium image exists”:

- inquiring whether the proportion of the number of pixels, which are in skin color and which are within predetermined ranges, of the detected thumb upward pattern to the number of pixels of thumb upward pattern exceeds a predetermined value,

- passing to the step of “taking at least one medium image from an image sensor which captures the images of the medium where the user exists” if the calculated proportion does not exceed the predetermined value.

BRIEF DESCRIPTION OF THE FIGURES

In Figure 1 , a representative view of the user interface control system is given.

In Figure 2a, a representative view of the medium image is given.

In Figure 2b, a representative view of the medium image is given.

In Figure 2c, a representative view of the medium image is given.

In Figure 3a, a representative view of thumb upward pattern is given.

In Figure 3b, a representative view of thumb upward pattern is given.

In Figure 3c, a representative view of thumb upward pattern is given.

In Figure 3d, a representative view of clicking movement is given.

In Figure 4, a schematic view of the user interface control system is given.

In Figure 5, a schematic view of the method steps is given.

DETAILED DESCRIPTION OF THE INVENTION

In this detailed description, the subject matter is explained with references to examples without forming any restrictive effect only in order to make the subject more understandable. The present invention is a user interface control system (100) which provides controlling of a menu displayed in a display unit (200) by means of body movements. Accordingly, basically, an image sensor (300) receives the medium images (310) of the medium where the user exists and a processor unit (110) processes these images and provides the user to move between the menu items by means of the hand movements of the user and provides the user to make selection.

Here, the mentioned display unit (200) can be a screen, projector, etc. The image sensor (300) can be a camera.

With reference to Figure 1 and Figure 4, the display unit (200) is associated with a processor unit (110). The processor unit (110) can be a microprocessor, etc. The processor unit (110) is associated with a memory unit (120) in a manner realizing reading and writing. The memory unit (120) can comprise memories arranged to keep the data in a permanent and temporary manner. The processor unit (110) is configured to control the image unit. In other words, the processor unit (110) provides displaying of a user interface for realizing input by the user in the display unit (200) and for getting output in correspondence, in other words, for interacting with the display unit (200). The user interface can comprise a menu. The menu comprises menu items. A marker provides marking and selection of menu items. When the menu item (230) is selected, the function of the menu item (230) selected by the processor unit (110) is realized. If the menu item (230) is marking process, it describes the condition where the user visually distinguishes a menu item (230) from the other menu items. This marking can be realized by means of a marker, by means of changing the color or the brightness of the menu item (230).

The memory unit (120) can comprise an interface control software (121) comprising functional modules formed by command lines executed by the processor unit (110). The processor unit (110) provides realization of user interface control by realizing the steps forming the innovative characteristic of the present invention by applying the commands of the interface control software (121).

In more details, the interface control software (121) comprises a face detection module (122), a thumb upward detection module (123), a confirmation module (124), a thumb upward monitoring module (125), an angle detection module (126) and a selection module.

The face detection module (122) receives a medium image (310) from the image sensor (300) and detects the faces which exist in the medium image (310). A rectangular area which encircles the detected face is defined as a face area (322). In this possible embodiment, face detection is realized by means of one of cascade classification methods. In more details, face detection is realized by using Haar Cascade classifiers. In Figure 2a, a medium image (310) is given where the face of a user is detected as in Figure 1.

The thumb upward detection module (123) searches the thumb upward pattern (330) of the user. As one of the innovative characteristics of the invention, it searches thumb upward pattern (330) in a scan area (321) with predetermined dimensions and which exists in the vicinity of the detected face. Thus, all of the medium images (310) are not searched and the process sources are used less. In more details, an area, which extends to the right hand direction and left hand direction at an amount equal to the width of a face area (322) which encircles the face and which is in rectangular form and which extends downwardly at an amount equal to four times of the height, is searched in the scan area (321). In Figure 2b, the representative view of the scan area (321) is given. This search can be realized by means of Flaar Cascade classifier. This classifier has been obtained as a result of realizing training by using different sub-clusters of the pluralities of positive and negative (With Thumb Upward Movement and Without Thumb Upward Movement) photographs. The classifier parameters are kept in a file with extension xml. Thus, since precise detection of the face is much safer, the accuracy of the detection of the thumb upward pattern (330) is increased. In Figure 2c, the representative view of a medium image (310), where the thumb upward movement exists, is given.

The confirmation module (124) confirms whether the thumb upward pattern (330) belongs to a hand, to a person. Thus, the confirmation module (124) prevents erroneous detection of items like a thumb upward pattern (330), poster, etc. In more details, it is controlled whether the lower half of the thumb upward pattern (330) has skin color or not. The skin colors can be recorded in a skin color database or they can be removed from the detected face. The thumb part of the thumb upward pattern (330) is in the upward direction and the hand part thereof is in downward direction. Accordingly, it is inquired whether the part, which remains under an imaginary line drawn horizontally from the center of the thumb upward pattern (330), matches with the skin color or colors at a predetermined proportion. In Figure 3b, the separated parts can be seen. In this possible embodiment, the proportion is 80%. Thus, erroneous results are not obtained due to accessories fixed to the hand.

One of the two color spaces can be used for color control. The first space is YCbCr space and it is inquired and determined whether a pixel is in skin color and whether it is within the below mentioned ranges: • 210 >Cb > 140

• 130 >Cr > 90

The second space is HSV space and it is inquired and determined whether a pixel is in skin color and whether it is within the below mentioned ranges:

20 > H > 0 255 > S > 48 255 > V > 80

In a possible embodiment of the present invention, as the additional or equivalent confirmation method, by determining the maximum dimensions which are to be obtained in the medium image (310) when an individual realizes thumb upward pattern (330) and the dimension of the detected face and the dimensions of the thumb upward pattern (330), it is detected whether the thumb upward pattern (330) is within these limitations. In more details, in bigness control, it is detected whether the thumb upward pattern (330) is compliant to the below mentioned limitations.

0.4 ^* face width < Thumb upward pattern (330) width < 0.6 ^* face width 0.8 ^* face height < Thumb upward pattern (330) height < 1 .2 ^* face height

The thumb upward monitoring module (125) detects the thumb upward pattern (330) in the sequential medium images (310) as the thumb upward pattern (330) is detected. In this step, face detection is not realized in sequential medium images (310). When the mode where the thumb upward pattern (330) is monitored is realized, preferably menu items are displayed in the displaying unit (200).

For monitoring the upward movement of the thumb, the lower part of the thumb upward pattern (330) is focused, in other words, the part where the hand is in fist form is focused. The lower part of the thumb upward pattern (330) is convex and there is less amount of background images which will affect the monitoring in an unfavorable manner. Various algorithms (for instance, KCF algorithm) can be used in the monitoring mode. When the thumb upward pattern (330) cannot be detected, the user returns to the face detection step.

The angle detection module (126) detects the finger lines (411) defined as lines between fingers which contact each other in the thumb upward pattern (330) and determines the angles thereof with respect to a predetermined reference axis (Figure 3a). In a possible embodiment of the present invention, the average of all finger angles is calculated. One of the menu items is marked according to the calculated angle. The finger lines (411) can be detected by using Hough transformation.

The selection detection module (127) rotates the thumb upward pattern (330) such that the thumb faces upwardly. After rotation, the pixels which exist in the skin color from upwards towards downwards are counted. When pixel decrease is detected in the skin color at the upper part of the thumb upward pattern (330), it is assumed that the clicking movement (350) is realized and the marked menu item (230) is selected. In Figure 3d, the representative view of the clicking movement (350) detected on the medium image (310) is given.

The steps applied by the modules are given in Figure 5 in a schematic manner.

In case more than one face is detected, thumb upward movement is searched separately for each face, and processes are realized accordingly.

The exemplary operation scenario of the present invention is as follows:

The user gets opposite to the display unit (200). The display unit (200) takes the medium image (310) of the user. The processor unit (110) detects the face from the medium image (310). The processor unit (110) realizes face detection in the sequential medium images (310) until it detects the thumb upward movement. The menu items are displayed on the screen. The processor unit (110) passes to the thumb upward movement monitoring mode (hand monitoring). Finger lines (411) are detected in another received medium image (310), and a screen item is marked on the screen according to the stop angle of the finger lines (411). Until it is detected that the user realizes selection sign, the thumb upward movement is detected and the angle is calculated and the menu items are marked. When the thumb upward movement of the user is detected, the last marked menu item (230) is selected and the function of the menu item (230) is realized.

The protection scope of the present invention is set forth in the annexed claims and cannot be restricted to the illustrative disclosures given above, under the detailed description. It is because a person skilled in the relevant art can obviously produce similar embodiments under the light of the foregoing disclosures, without departing from the main principles of the present invention. REFERENCE NUMBERS

100 User interface control system 110 Processor unit

120 Memory unit

121 Interface control software

122 Face detection module

123 Thumb upward detection module

124 Confirmation module

125 Thumb upward monitoring module

126 Angle detection module

127 Selection detection module 200 Display unit

210 User interface 220 Menu 230 Menu item 300 Image sensor 310 Medium image

320 Face

321 Scan area

322 Face area

330 Thumb upward pattern

331 First region

332 Upper part

333 Lower part

334 Reference line

335 Reference point 350 Clicking movement 400 User

410 Finger

411 Finger line