This disclosure relates to the control systems of devices that are out of user’s direct vision, in particular to the systems for manual control of a handheld device that is out of user’s direct vision.

State of Art

A smartphone is an integral part of major people's lives. People use it for work, entertainment, communication, information reception or just to "kill time". It becomes a habit to constantly have a phone "at hand", or rather, to always be able to see information on its screen and use it. This habit usually occurs in situations when a person has more free time. In most cases, the phone is accessible to a person, however there are circumstances or situations when it is very complicated or even impossible to use the phone. For example, during a visit to the dentist. When a dentist examines or treats a patient, the latter usually does not have access to the phone, since various dental equipment and tools, as well as the staff interfere with the direct vision of phone screen (in the case of smartphones - they can usually be applied by touching the screen with the fingers and with a need to see both the screen and one's fingers). The ability to use a smartphone during dental treatment or other procedures would allow the patient to make effective use of his/her time, and the shift in focus from the treated area of the body to the contents of the phone will make the patient experience easier.

The patent document US2019265488A1 (publication 2019-08-29) provides a head mounted display system that comprises a main body configured to be worn on a human head and to secure therein a mobile computing device with a display and hardware for generating images on the display corresponding to motion data. The other patent documents CN206960779U (publication 2018-02-02) and CN207473212U (publication 2018-06-08) provide virtual reality glasses that comprise a frame for mobile phone placing, and the contents of mobile phone screen is displayed to the user. However, the user cannot touch the screen with his/her fingers and thereby control the mobile device (phone).

The patent document US10530914B2 (publication 2020-01-07) provides a head-mounted display system and method of operation that allow users to couple a portable electronic device such as a mobile phone with a separate head-mounted device (glasses). The patent document GB2405049A (publication 2005-02-16) provides the spectacles with integral hands-free kit for a remote unit such as a mobile phone or computer. The other patent document US2007037520A1 (publication 2007-02-15) provides the glasses that comprise microphone, transmitter, speaker, receiver, and power source, for sending and receiving signals wirelessly to and from a remote mobile phone or other electronic device.

The closest patent document US10338673B2 (publication 2019-07-02) provides a system that comprises a head mounted electronic device including a display, a camera and a processor, the head mounted electronic device configured to be operably coupled with a handheld electronic device (smartphone), wherein the camera is configured to capture an image of the handheld electronic device, and the processor is configured to segment an image of a physical object detected between the head mounted electronic device and the handheld electronic device, including the virtual image of captured object and the image of handheld electronic device interface, and to display them on the screen of head mounted device. A hover position of user's fingers at a handheld electronic device is detected by rendering / overlaying monochromatic screen, for instance, the green screen and using the camera integrated into the head mounted device. The user interface of the handheld electronic device may be virtually rendered and displayed on the screen of the head mounted device along with the display of the fingers' position of handheld device user on a monochromatic screen of the handheld device. In such a way the user may be able to physically interact with the user interface of the handheld electronic device and make use of its functionality even though the user interface is not visible to the user, but instead is virtually displayed on the screen of the head mounted device. The user, for instance, can be rendered with a virtual keyboard on the screen of the head mounted device, detecting the fingers' position on a monochromatic screen of the handheld device and at the same time displaying the virtual image of the fingers on the screen of the head mounted device.

The technical solution described in the disclosure characterized in that a smart device (e.g. smartphone) is placed next to a holder equipped with camera coupled with a device mounted in front of the user's eyes, in the screen therein (e.g. in VR glasses) the user can see a real image of his/her smart device screen (e.g. smartphone) as well as his/her fingers operating such smart device. Thereby the user can use his/her smart device (e.g. smartphone) as usual, i.e. viewing his/her fingers and the device screen, and, contrary to the closest patent document, the real images of the smart device screen and user's fingers are displayed; the smart device screen rendered / overlaid by monochromatic screen is not required for detection of finger position, and the detection of physical objects between the head mounted device and smart device is not needed. Also, no additional training is required in the use of user interfaces and the control system of virtual reality glasses or other similar devices, and there is no need to install additional software on a smart device. In addition, the system described in the disclosure ensures the privacy of the smart device user since no additional software is required, and with addition of protective element, such as a hood, unauthorized persons cannot see the contents of the user's smart device screen. Furthermore, the system is of closed type as it does not record or transmit the image (including screen content) of the user's smart device to others, although it can do so if necessary. The described solution also allows using other tools that are out of direct vision of the user, for example, reading a book, taking notes, drawing sketches, etc.

Summary of the Invention

For the purposes of this disclosure, the term "handheld device" involves all types of devices that require vision of the screen and the user's fingers for operation, including smartphones, tablets, and other electronic devices with touchscreen, also push-button phones and electrical, mechanical equipment of similar size, as well as the units made of any material that require vision of the unit and human fingers for operation or usage. The disclosure will provide examples related to the use of smart handheld device such as a smartphone, however it does not limit the application of the system to control or use other devices or objects (e.g. book reading).

For the purposes of the description, the term "display device" involves all types of display units and screens designed to display an image to the user, including VR glasses, head mounted, as well as similar devices directly mounted in front of or into user's eyes (video contact lens) and capable of transmitting visual image to the user's eye retina.

For the purposes of the description, the term "power source" involves all types of stationary and portable power supply sources including alternative energy such as solar or wind, as well as power sources capable of wireless power transfer.

The technical solution described in this disclosure is a system for manual control by user of a device that is out of user's direct vision. The above system comprises at least: a holder where at least one video camera attached to, the holder can be placed or installed on various objects, a size of the holder is fixed by rigid joint or adjustable by sliding joint; a handheld device placed next to (under) the camera; a display device coupled with video camera to display the image of the handheld device (in the case of smartphone - its screen and the image of user's finger position); at least one processor connected between camera and display device and applied only when the compatibility of camera and display device interfaces are required. The camera, display device and image processor, when the latter is in use, are interconnected by physical connection or wireless connection.

Depending on embodiment examples the system additionally comprises: at least one power source to feed video camera, display device and image processor, where the said power source can be portable and/or stationary, and additionally can include an indicator showing the charge level of the power source and warning the user about the necessity of power source charging; a protective element to cover the handheld device and, if needed, the camera from prying eyes, surveillance cameras and similar devices, and against physical impact; a retaining element of handheld device to secure the handheld device in stable position subject to camera or for other purposes, where the said retaining element can additionally include a hand and/or finger support to comfortably hold and / or operate the handheld device; a holder locking element to fix camera to user's hand or other part of the body, or other nearby object; a control light to inform the user that camera is on and transmitting the image to the display device; a switch integrated in the holder or in protective element to turn on/off the video camera and/or display device, and/or image processor, if the latter is applicable; an adjustment device for camera and/or image processor settings.

Brief Description of the Drawings

Fig. 1. Layout scheme of system for manual control of a device out of user's direct vision.

Fig. 2. Flowchart of system for manual control of a device out of user's direct vision.

Description of the Preferred Embodiments

The system described in this disclosure is designed for manual control of handheld device that is out of user's direct vision. The system is applied when the user for any reason cannot see a handheld device and therefore cannot operate or use it properly (for example, in case of smartphone usage when the user cannot see its screen and access the contents therein, and when for smartphone operation it is necessary to see the screen and the position of his/her fingers on it). The mentioned system comprises at least:

- a holder (10), where at least one video camera attached to (30),

- a handheld device (20), placed next to (under) camera (30), - display device (40), coupled with video camera (30), to display the image of the handheld device (20) (in the case of smartphone - its screen and the image of user's finger position),

- at least one image processor (60), connected between camera (30) and display device (40), and applied only when the compatibility of camera (30) and display device (40) interfaces is required.

The system additionally comprises:

- at least one power source (50) to feed video camera (30), display device (40) and image processor (60).

- a protective element (103) to cover the handheld device (20) and, if needed, video camera (30) from prying eyes and physical impact,

- a retaining element of handheld device (20) to secure the handheld device (20) in stable position subject to camera or for other purposes,

- a holder (10) locking element to fix camera (30) to user's hand or other part of the body, or other nearby object,

- a control light to inform the user that video camera (30) is on and transmitting the image to the display device (40),

- a switch integrated in the holder (10) or in protective element (103) to turn on/off the video camera (30) and/or display device (40), and/or image processor (60), if the latter is applicable,

- an adjustment device for camera (30) and/or image processor (60) settings.

The said holder (10) comprises at least one section (101), where video camera (30) is attached to, and holding section (102) to fix or place the holder (20) on various objects, wherein both sections (101, 102) can be interlinked by rigid joint or sliding joint enabling holder (10) sizing. The holder (10) can be fixed to or placed on the body of handheld device's (20) user, for example, on the belly, leg (-s), wrist or other parts of the human body, as well as on the nearby objects, for example, chair, table or similar objects. The handheld device (20) is placed under video camera (30) so that the field of vision should cover the image of the device (20) (e.g. smartphone, including its screen, and image) and user's fingers operating or using the device.

A protective element (103) can be added to the holder (10) to cover the handheld device (20) and, if needed, video camera (30) from prying eyes and/or surveillance cameras or similar equipment, and/or physical impact, e.g. spray, splashes, lamp reflectors or similar impact. The protective element (103) can cover the entire structure of the holder (10) or only a part of it, for example, the upper part.

A retaining element of handheld device (20) can be mounted to the holder (10) in order to secure the smart device (20) in stable position subject to video camera (30) or for other purposes. The said retaining element can additionally include a hand and/or finger support to comfortably hold and/or operate the handheld device (20).

Also the holder (10) can be equipped with a holder locking element to fix camera (30) to user's hand or other part of the body, or other nearby object.

The holder (10) can be connected to the at least one power source (50) to feed camera (30), display device (40) and image processor (60), when the latter is applicable. The power source (50) can additionally feed the handheld device (20), when the latter is an electrical device.

The holder (10) or protective element (103) can additionally include a control light to inform the user that the video camera (30) is on and transmitting the image to the display device (40), and a switch to turn on/off the video camera (30) and/or display device (40), and/or image processor (60), if the latter is applicable.

The video camera (30) is applied to capture the image of handheld device (20), for example, smartphone screen, and the image of user's fingers operating or using the handheld device (20), and render such images to the screen of display device (40). The camera (30) can additionally include a plug for physical connection, linking it with display device (40), video camera (30) or image processor (60), when the latter is applicable. However the above elements can be provided with wireless connection, without physical link.

A display device (40) comprises at least a screen to display the image rendered by video camera (30). The display device (40) can additionally include a plug for physical connection, linking it with display device (40), video camera (30) or image processor (60), when the latter is applicable. However, the display device (40) with video camera (30) or image processor (60), when the latter is applicable, can be provided with wireless connection, without physical link. The display device (40), if consisting of VR glasses, can additionally include at least one lens positioned in the housing of display device (40) in front of at least one eye of the user. The display device (40) in such a case is mounted on the top of the head in front of user's eyes using a fastener (401), and camera (30) image is transmitted to user's eyes. The display device (40) can also be distant from the user and can be placed in other positions at the user's eye level, for example, on the wall or in special holder, or on the user's eye such as a digital contact lens or similar device. At least one image processor (60) is applied when the compatibility of camera (30) and display device (40) interfaces is required. The said image processor (60) can process the image of handheld device (20) received from video camera (30) (in the case of smartphone - its screen and the image of user's finger position), and render it to the display device (40), if such functions are not performed by camera (30). The image processor (60) is connected between camera (30) and display device (40). Therefore, the image processor (60) can additionally comprise at least two plugs for physical connection of image processor (60) with camera (30) and display device (40). However, the said elements can be wireless interconnected, without plugs, or using one plug if physical connection to one of devices (30 or 40) is applied. The image processor (60) can perform more functions such as changing the focus of camera image, adjusting the exposure, and making other image adjustments.

At least one power source (50) to feed video camera (30), image processor (60), when the latter is applicable, and display device (40), if the latter is without its separate power source. The power source (50) can be stationary and/or portable. The power source (50) can be installed in the holder (10) or any other place. When the said power source (50) is portable, it can additionally include an indicator showing the charge level of the power source (50) and warning the user about the necessity of power source charging (50).

The system can be additionally provided with a device for settings adjustment of camera (30) and/or image processor (60), when the latter is applicable, such as exposure, focus, brightness and etc.

The system functioning comprises the following steps:

- when the user is unable to directly see a device held in his/her hands, for example, at the dentist during dental treatment procedure, a smartphone is placed next to a holder (10) under video camera (30), where the holder (10) is located at a distance from the user, which allows the user to easily and comfortably reach the handheld device (20) with stretched one hand or both hands;

- when the solution of display device (40) with VR glasses is applied, a display device (40) is placed on the user's forehead in front of eyes, and is connected with video camera (30) or image processor (60), if the latter is applicable. Other solutions are also available: for example, the image of the handheld device and user's fingers is displayed on the display device that is distant from the user, but in a clearly visible view; - the user or assisting person turns on the camera (30), display device (40) and image processor (60), if the latter is applicable. When the control light is on, it informs the user that video camera (30) is activated and transmitting the image to the display device (40);

- the video camera (30), secured to the holder (10), and positioned to encompass the image of handheld device (20) placed next to the holder (10), for example, the images of smartphone screen and user's fingers operating the said smartphone (20), renders the image to the image processor (60), which making camera (30) and display device (40) interfaces compatible, further transmits the images to the display device (40). As a result the user is able to see on the screen of display device (40) the screen image of his/her handheld device, e.g. smartphone, as well as finger position, and can operate and use the handheld device. The user can also, by means of additional device, adjust the settings of camera (30) and/or image processor (60), when the latter is applicable, such as exposure, focus, brightness and etc.

In order to illustrate and describe the present invention, the above is a description of the most preferred embodiments. This is not an exhaustive or limiting description intended to determine the specific form or embodiment example. Obviously, many modifications and variations will be apparent to those skilled in the art. An embodiment is selected and described for those skilled in the art to better understand the principles of the present invention and their best practices for various embodiments with different modifications suitable for a particular use or application of the embodiment. It is intended that the scope of the invention be defined by the accompanying claim and its equivalents, in which all of the above terms have their broadest meaning unless otherwise indicated.

The embodiments described by those skilled in the art may be subject to modifications within the limits of the scope of the present invention as defined in the claim below.