Field of the invention

The invention concerns methods for providing remote access to stationary computers via mobile phones. Background of the invention

Remote operation of computers is a well-known concept. Today, there are serveral communication channels that can be used to connect to and operate a computer at short or long distances, including the phone network, cellular phone networks, local area networks (LANs) in offices or buildings, wireless networks (Wi-Fi), and the Internet. Hereafter, we will denote the computer that is being remotely operated the "host", and the device that is used to perform the remote operation the "terminal".

There are a multitude of software products available that will enable remote operation of a Windows PC, with a graphical user interface, over one or several of the communication channels mentioned above. The most intuitive operation of the host becomes possible when the terminal also is a PC. In this case, the full display image from the host can be shown on the display of the terminal, and mouse and keyboard input at the terminal are sent to the host. Except for the possibility of slight delays due to bandwidth-limited communication, the user experience is that of sitting directly at the host with access to all its software, processing power and peripheral equipment.

In many situations, remote operation of a host from a mobile terminal is desirable, for example when a person is on a business trip and wishes to access documents or software on the office computer. In particular, the utilization of a mobile phone as a terminal is of great interest, since today it is very common to carry a mobile phone at all times. A convenient way of operating a remote computer from a mobile phone would be a very significant extension of the phone's capacity, enabling access to a powerful processor, large internal and external storage capacity, a full range of software including such applications that put high demands on processor and internal/external memory, and frequently also a high-speed Internet connection. In brief, the power of a desktop computer would in many aspects be transferred to the mobile phone.

Remote operation from wireless terminals is possible from laptop PCs with Wi-Fi network capability or an internal or external wireless modem, the latter typically being a mobile phone with data transmission capability. Drawbacks of this kind of solution are that you have to carry both a laptop and a mobile phone, and that it is not convenient to open up and operate a laptop anywhere. There are also products on the market that enable remote operation from personal digital assistants (PDAs). These solutions are more mobile than the laptop concept, but suffer from the lack of full-size keyboard and mouse. Further, PDAs do not have large enough displays to be able to show the full host display image. This problem has been addressed by providing a scrolling function, scaling the host display image to the size of the PDA display at the cost of lost detail, and by methods such as disclosed in US patent No. 6,710,790, where the PDA display is automatically scrolled to the region of interest if an event (such as the appearance of a pop-up dialog window) occurs in the host that causes the display image to change outside the region which is currently shown on the terminal display.

Remote PC operation from ordinary mobile phones is not commercially available. The drawbacks present in PDAs are even more pronounced in mobile phones, since they usually have even smaller displays, only a numerical keypad, and no pointing device. The only exception known to the applicant is a remote operation product for the Nokia Communicator mobile phone, which is a PDA-style phone with a full keyboard. In this product, the pointer/cursor control occurs via the four- way control pad, by which the pointer is moved to its desired position by pressing and holding the control pad until the pointer has moved far enough. This way to control the pointer has a major drawback in that it will be very slow compared to ordinary mouse operation. Further, it is less intuitive than using an ordinary mouse. Another drawback is that display scrolling occurs in the same way, using the four-way control pad, with the same disadvantages as for pointer control. Other available ways of remotely accessing office PCs from mobile phones include various applications that provide access to certain applications and/or datasets in the host, such as e-mail, contact list, calendar, etc. Although useful for their intended purposes, such products can not provide full access to the full scope of software, data, processing power and peripheral equipment present on the host.

In summary, remote PC operation from ordinary mobile phones is is associated with a number of obstacles, including the inherently small display size compared to the host, the lack of a mouse or a cursor control device of similar capacity, and the lack of a fully equipped keyboard. Today, a large fraction of buyers of new mobile phones choose models that are equipped with digital cameras, and the fraction is increasing. The applicant has filed a patent application for an information processing system and method, where an optical image recorder (e.g. a digital camera) in a handheld device (e.g. a mobile phone) is employed to track the motion of said device and where the so obtained motion parameters are used as input to the control electronics of said device, enabling improved functionality such as display scrolling, cursor movement and menu selection. The application (Norwegian application No. 20044073) had not been published at the time of filing of the present application. Summary of the invention

The object of the present invention is to provide a method for conveniently operating a remote computer from a mobile phone, using movement of the phone as a vital component in the interaction with the remote computer, and building in part on the invention in the applicant's previous patent application.

This object, as well as further features and advantages, is realized with an information processing system comprising: a mobile phone (the terminal), comprising a colour display, an internal memory into which data and program code can be downloaded, a processor capable of executing program code in the internal memory, input/output capabilities, capacity to transmit data over a cellular phone network, and further comprising a digital camera or built-in gyroscopic or accelerometer sensors, a computer (the host) with a graphical user interface (GUI) and being permanently connected to the Internet, and characterized by said mobile phone comprising software capable of analyzing signals from said camera or sensors to yield phone movement vectors, which are converted to represent mouse movement vectors and/or display scrolling vectors that are transmitted to said host,

and further characterized by said computer comprising software which receives said mouse movement vectors and utilizes them to control the cursor movement on said computer, and which also receives the display scrolling vectors and utilizes them to calculate the coordinates of the subsection of the computer display that is currently shown in the display of said mobile phone, and which transmits data that describes said display subsection to said mobile phone where it is used to draw the subsection of the computer display image, and still further characterized by the possibility to utilize said phone movement vectors to move a cursor across a virtual keyboard on the display of said phone, thus producing the equivalent of keyboard input, which is then transmitted to said computer where it replaces input from the local keyboard.

Further features and advantages shall be apparent from the appended dependent claims.

The invention shall now be explained in more detail by resorting to a discussion of exemplary embodiments thereof and with reference to the accompanying drawing figures, in which

Fig. 1 shows the host display with examples of three different locations of the mouse pointer and of the display subsection that is shown on the terminal. In the first case (1), the pointer and subsection are free to move in any direction and the subsection is centered around the pointer coordinates. In the second case (2), the pointer has been moved so that the subsection has reached the right border of the host display. In the third case (3), the pointer has been moved downward and even further to the right, and the subsection has followed downwards but not further to the right, fig. 2a a mobile phone which is currently used for text (keyboard) input in a remote operation situation. A virtual keyboard (1) has been invoked on the lower part of the phone display, and the text insertion cursor (2) is shown in the center of the display, fig. 2b shows the appearance of the virtual keyboard during input of the ctrl+c key combination: The highlight of the ctrl key remains until the c key has been selected, fig. 3a the front side of a mobile phone,

fig. 3b the back side, with a digital camera (1) positioned on the opposite side from the display, fig. 3c a user (2) holding the mobile phone (3) (seen from the side) such that the field of view (4) of the camera is approximately the same as the field of view (5) of the user, fig. 4 the host display (1) and an enlarged view of the host display image (2). A virtual menu bar (3) is added to the display image. This menu bar can have buttons (4), text menus (5) and pull-down menus (6). It will not be visible on the host display, but only in the terminal's display, and will provide added functionality that relates to the remote operation situation.

Common to all embodiments are the following basic principles for how the remote operation works:

A data transmission connection is established between the terminal and the host, using one or a plurality of the following communication channels: cellular phone network, ordinary phone network, the Internet, local area network (LAN), wireless LAN including Wi-Fi network, short-range radio frequency connection (e.g. Bluetooth), infrared communication. It is recommended, but not crucial for the present invention, that the connection is encrypted using e.g. the AES algorithm. The user confirms his identity and right to access the host by entering a password or by another method of identification.

The terminal sends data to the host indicating the size in pixels of the image that is going to be displayed on the terminals display. The host responds by sending an image, preferably in a compressed format, of an adequately sized subsection of its display, centered around the location of the mouse pointer.

When the terminal (i.e. the phone) is moved, the movement is detected and mouse movement vectors are generated internally in the terminal. These are sent to the host where they are handed over to the operating system in such a way as to be treated as representing movement of the mouse. In most cases, this means that the mouse pointer on the host display is moved to a new position.

When the mouse pointer is moved, a new display subsection position is calculated in the host and the data needed to update the terminals display is sent to the terminal.

Left and right mouse buttons are assigned to two buttons on the terminals keypad. Key press and release events are sent to the host where they represent operation of the mouse buttons.

When operating as outlined in the previous paragraphs, the display subsection will always be centered around the mouse pointer position. There are however cases when this restriction is lifted. First, if at least one of the subsection borders coincides with one of the borders of the host display, the subsection will not be moved any further in that direction and the pointer will become free to move in that direction until it reaches the border of the host display. This case is illustrated in fig. 1. Second, the user may send a command that freezes the scrolling function and allows the mouse pointer to move freely within the display subsection window. Third, the user may send a command that freezes the mouse pointer and allows scrolling freely across the whole host display. When done using the free scrolling, the user can send a command either to snap the subsection into place centered around the mouse pointer, or to snap the mouse pointer into place in the center of the current subsection.

Keyboard input from the terminal to the host is invoked in one of two ways. First, it is possible to use the standard text input concept in mobile phones, i.e., pressing one of the numerical keypad keys (usually 0-9, * and #) on the phone keypad one or several times to send a letter or another symbol. Second, a virtual keyboard can be invoked on part of the phone display (see fig. 2a). The mouse and scrolling functions are locked and if there is a text insertion cursor on the host screen, the display subsection is centered around this cursor. The phone movement is used to move a highlight symbol across the virtual keyboard, and one of the phone buttons is used to accept the currently highlighted key. Modification keys like ctrl, shift and alt are used prior to the key they are meant to modify, i.e. if the user wants to send ctrl+c to the host, he first highlights and accepts the ctrl key, then the c key on the virtual keyboard (see fig. 2b). The virtual keyboard can be semi-transparent to keep a partial view of the underlying host display. In the virtual keyboard mode, the numerical keypad keys are also still accepted for text input.

In a particularly preferred embodiment, the mobile phone is equipped with a digital camera positioned on the opposite side of the phone with reference to the screen, thus viewing approximately the same scene as the user if the phone is held in front of the user (fig. 3). Given this configuration, the movement of the phone in the left/right and up/down directions can be estimated by analyzing the images from the camera when operated in video mode, according to the description in our previous patent application (Norwegian patent application No. 20044073). The mouse pointer movement and/or display scrolling will be coupled to the phone movement in a highly intuitive way: If the user wishes to move the pointer to the right, he moves the phone to the right; if the user wishes to view a section of the host display- image that is to the right of the currently shown subsection, he also moves the phone to the right.

By scrolling the phone display image in a direction opposite to the phone movement vectors, the impression will be created that the phone display is a window on a larger image, which in the present invention is the host computer display image. The user can easily and quickly view any part of the host display by simply moving the phone around, which is a major improvement over currently available display scrolling methods. Many new mobile phones are in their standard configuration equipped with digital cameras that meet the requirements of the present embodiment. Thus, this embodiment is especially attractive since it provides a significant added functionality in a widely available platform without the need for any hardware replacement or modification. As described in Norwegian patent application No. 20044073, turning and/or tilting the phone will have the same effect on the scene captured by the camera as lateral movement of the phone; therefore, turning/tilting is a complementing mode of control which is especially suited for situations where there is limited space for moving the phone, or where the imaged scene is far from the user (in which case the scene will remain nearly constant if the phone is moved laterally).

In a second embodiment, the mobile phone is again equipped with a digital camera which is positioned and used in the same way as in the previous embodiment. In addition, the image analysis procedure is extended to extract also phone movement in the third dimension - in/out, i.e., towards or away

from the user. This added dimension provides additional functionality and is used to change the scale of the image shown on the phone display. Moving the phone towards the user will cause the size of the displayed subregion to increase relative to the size of the host display, which will be experienced as a zoom out operation. Naturally, there will then be no pixel-to-pixel correlation between the host and the terminal; instead the phone display will show a reduced-resolution image of a larger subregion. The zoom-out movement can be continued until an overview of the entire host display is shown; further movement will not result in any additional zooming. The zoom factor can be changed either continuously or stepwise; the latter putting less demand on the hosts processing power since zoom factors can be preselected that have easily calculated pixel correspondences (e.g. 2:1).

This embodiment provides a fast way to get an overview of the entire host display and, using the other two dimensions of phone movement, to coarsely move the mouse pointer to any position. A zoom-in operation by phone movement away from the user will gradually or stepwise increase the magnification until pixel-to-pixel correlation is achieved. Any intensive processing associated with these zoom operations will be executed by the host computer, avoiding excessive load on the processor in the phone, which in most cases has a far lower capacity.

In a variant of this embodiment, the zooming directions are reversed for users who prefer this, meaning that zoom-in will occur when the phone is moved towards the user, and zoom-out when the phone is moved away from the user. In a third embodiment, the phone is internally or externally equipped with gyroscopic or accelerometer-based motion sensors, providing information about the phone's orientation and movement in two or three dimensions. The utilization of this information is identical to the descriptions in the first and second embodiments, except that the need to use rotation/tilt will be smaller, since there is no dependence on the distance to the background scene.

In a variant of any of the three previous embodiments, a virtual menu bar is added to the host display image along any of the four edges (fig. 4). This menu bar is not part of, and does not infringe on, the actual host display image and is seen only in the phone display when scrolled to the appropriate edge. The menu bar holds buttons or pull-down menus that accommodate

functions that are useful in a remote operation situation, such as blanking of the host display, locking of the host keyboard and/or mouse, change of password, restart host computer, send ctrl+alt+del keyboard combination, etc. Generation of the virtual menu bar is handled by the host computer, avoiding extra processor load and memory requirements on the phone.