Augmented visual assistance system for assisting a person working at a remote workplace, method and headwear for use therewith

Field of the invention

The present invention relates to the remote instruction or training of people in practical skills on the job rather than in the classroom, and headwear for facilitating such training.

Background

The demographics of the population is aging and in some industries the number of long-experienced workers is declining. Accordingly, it is not always possible or convenient to have an experienced service worker or skilled technician available to perform a task, particularly at a remote location. These experienced workers or technicians are referred to herein as "knowledge providers".

Also, at times a person, which may be an unskilled or semi-skilled worker, is faced with tasks that they are not trained to deal with or are unsure about and there is a need for that person to receive guidance to complete the required task. It is not always possible or practical to have an experienced service person available on site to guide the worker.

Preferred embodiments of the invention can provide a way to efficiently provide assistance and/or training from more experienced people to people such as workers, who may be semi-skilled, in an efficient way. People receiving assistance and/or training are referred to herein as "knowledge acquirers". Preferred embodiments of the invention can also provide a new way of imparting skills or training remotely.

Summary

There is provided a system for providing visual assistance to a knowledge acquirer working at a workplace remote from a knowledge provider, said system comprising: headwear including :

a forward-facing camera for obtaining a moving image of a view of the workplace as seen by the knowledge acquirer and converting it to a first video electric signal; a microphone positioned for detecting the voice of the knowledge acquirer and converting it to an audio electric signal;

speakers for producing audible instructions for the knowledge acquirer;

a projector for projecting an overlay image onto the workplace so as to at least partially overlap the moving image of the workplace captured by the camera; a controller for:

receiving and processing said first video electric signal and said audio electric signal before passing said processed signals to a communications module, the communications module being adapted to wirelessly communicate the processed signals to said knowledge provider;

receiving and processing an incoming voice signal from said communications module and directing said processed incoming voice signal to said speakers; and receiving from said communications module a second electric signal containing graphic information, processing said second electric signal and passing said processed signal to the projector, wherein the communications module is also adapted to wirelessly receive said incoming voice signal and said second processed electric signal from the knowledge provider; and

a source of electric power connected to said camera, said speakers, said controller, said communications module and said projector; and

a remote computing unit configured for viewing by the knowledge provider the first video signal and for receiving input instructions from the knowledge provider, the input instructions being converted to the second electric signal and communicated to the projector via the communications module for display at the worksite by the projector.

Preferably, the communications module is a mobile phone running a mobile application (App) and is in communication with a mobile data network, the App being configured for transmitting data between the phone and the remote computing unit via the internet. The communications module may be also part of the headwear where a separate mobile phone is not permitted.

The headwear can include a central control unit, batteries, headphones, microphone, camera, projector and a light source positioned on the headwear such that weight thereof is evenly balanced on a user's head when a headband is adjusted to fit the user's head. Preferably, the headwear includes a hard shell suitable for protection against impacts.

There is also provided a headwear for use in a system of the above described type.

There is also provided a method of assisting a knowledge acquirer using a system of the above described type at a workplace remote from a knowledge provider operating a computing unit, said method comprising:

obtaining from said camera a moving image of a view as seen by the knowledge acquirer and converting said moving image to a first video electric signal,

obtaining from said microphone an audio electric signal,

by said controller:

receiving and processing said first video electric signal and said audio electric signal before passing said processed signals wirelessly to a communications module, and

wirelessly receiving and processing an incoming voice signal and a second electric signal containing graphic information from said communications module and directing said processed incoming voice signal to said speakers and said second electric signal to the projector, and

by said knowledge provider:

viewing the video electric signal on the computing unit and using an input device to input instructions to the computing unit, the instructions being communicated from the computing unit to the communications module for display at the worksite via the projector.

The input device can be a mouse, the mouse being used to add predetermined symbols or graphical representations to a display on the computing unit so as to overlay the image. The input device can also be used to draw freehand lines on the image. There is also provided a headwear for instructing or training a person working at a workplace remote from a knowledge provider person, said headwear comprising :

a forward-facing camera for obtaining a moving image of a view of the workplace as seen by the knowledge acquirer and converting it to a first video electric signal; a microphone positioned for detecting the voice of the knowledge acquirer and converting it to an audio electric signal;

speakers for producing sounds adjacent at least one ear of the knowledge acquirer; a projector for projecting an overlay image onto the workplace so as to at least partially overlap the moving image of the workplace captured by the camera; a controller for:

receiving and processing said first video electric signal and said audio electric signal before passing said processed signals to a communications module, the communications module being adapted to wirelessly communicate the first processed video electric signal and said processed audio electric signal to said knowledge provider, and to wirelessly receive said incoming voice signal and the second processed video electric signal from the knowledge provider;

receiving and processing an incoming voice signal from said communications module and directing said processed incoming voice signal to said speakers;

receiving from said communications module a second electric signal containing graphic information and passing said processed signals to the projector, and a source of electric power connected to said camera, said speakers, said controller, said communications module and said projector.

Preferably, the headwear has a hard shell suitable for protection against impacts.

Brief description of the drawings

In order that the invention may be more easily understood, an embodiment will now be described, by way of example only, with reference to the accompanying drawings, in which :

Figure 1: is a front view of headwear according to a first embodiment of the invention;

Figure 2: is a side view of the headwear in Figure 1;

Figure 3: is a sectional view the status indicators lights for viewing by the knowledge acquirer in an upper corner of their eyes.

Figure 4: is a schematic drawing showing the interaction between a wearer of the headwear and a knowledge provider.

Detailed description

A headwear 10 according to a preferred embodiment of the invention is shown in

Figure 1. The headwear 10 is in the form of a safety hat and configured for use with a system for training or instructing a knowledge acquirer working at a workplace remote from a knowledge provider person. The hat 10 has as its basis a standard hard-shell industrial safety hat 11 as worn on construction sites, though it will be appreciated that it may take other forms such as semirigid headwear or a soft cap, for use in workplaces not requiring head protection.

Upon the hat 10 is mounted a camera 14, LED lights 16, a speaker 18 mounted near each ear, a microphone 20, a projector 21 and a controller and processor unit 22. A battery pack 24 is provided within an upper internal portion of the hat and contains a rechargeable battery.

The components incorporated into the hat 10, i.e. the camera 14, LED lights 16, speakers 18, microphone 20, projector 21, controller and processor unit 22, and battery pack 24 are positioned on the headwear such that weight thereof is evenly balanced on a user's head when a headband is adjusted to fit the user's head. The hat 10 may be configured to be of light weight construction. This may be achieved through selection of the components mentioned above, or through the construction of the hat 10. The camera 14 is preferably a CCD colour camera which is forward -facing and configured for obtaining a moving image of a view of the workplace as seen by the knowledge acquirer. In the examples provided, the knowledge acquirer is a worker who is a semi-skilled technician wearing hat 10. Hat 10 obtains a moving image of a view of the workplace and converts it to a first video electric signal. The camera 14 is centrally mounted in a brim of the hat 10 so as to be directly above a point which is between the eyes of the semi-skilled technician. The camera 14 may be independently adjustable, i.e. zoom and tilt, and is preferably mounted within the confines of the hat 10 so as to be protected from damage and may be configured for taking still photographs. A led light 16 is provided to illuminate the workplace, to improve the vision of the semi-skilled technician and enable the camera 14 to capture a clear video signal.

The microphone 20 is configured for detecting the voice of the semi-skilled technician and converting it to an audio electric signal to enable the semi-skilled technician to communicate with a knowledge provider. The microphone 20 is preferably mounted to a forward portion of the hat 10. The microphone may also be used in a voice command system for controlling the hat 10 as illustrated in figure 3 item 20.

The speakers 18 are configured for producing sounds audible to the wearer so that verbal instructions may be relayed to the semi-skilled technician. The speakers are preferably fitted to a side of the headwear, close to a wearers ear. A single speaker or multiple speakers may be used. The speakers 18 allow the knowledge provider to communicate with the semi-skilled technician. Also, for noisy environments, earphones or headphones can be provided. The earphones may be earbuds mounted on a retractable cable to keep the earbuds safely kept inside the hat 10 when not in use. Also, a 3.5mm headphone jack may be provided instead to enable standard headphones or ear buds to be inserted directly into the technician's ear canal.

The projector 21 is configured for projecting an image onto the workplace to partially or fully overlap the moving image of the workplace captured by the camera. This allows further communication with the semi-skilled technician from the knowledge provider and, in use, a knowledge provider at a remote location to draw the attention of the semiskilled technician to a particular point on seeing the video electric signal from the camera 14. As will be described in further detail below, the projector allows display of an image onto the workplace so as to either communicate information to the semi-skilled technician or to draw their attention to an object or position.

The controller (A/V processor 22) is configured for receiving and processing the first video electric signal obtained from the camera 14 and the audio electric signal obtained from the microphone 20, before passing those processed signals to a communications module (not shown). The communications module is mounted within the shell of the hat 11 and is adapted to wirelessly communicate the first processed video electric signal and said processed audio electric signal to the knowledge provider, and to wirelessly receive said incoming voice signal and the second processed electric signal from the knowledge provider. The controller is also configured for receiving and processing an incoming voice signal from the communications module and directing the processed incoming voice signal to the speakers 18. The controller is also configured for receiving from the communications module a second electric signal containing graphical information and for passing the processed signals to the projector 21.

Figure 4 illustrates the hat 10 within the described system 70. The communications module 42 of the hat 10 is configured for communication with a mobile device 44 such as a mobile phone or tablet, preferably using standard protocols such as Bluetooth for example. The mobile device 44 should have the ability to operate in standard mobile or cell communication protocols and frequency bands, such as the 3G, 4G or 5G ranges for example. It will be appreciated that communication via other radio signals are also possible, particularly in worksites with infrastructure to support such communication.

The mobile device 44 is used to transfer the first and second electric signal and voice signals over the Telecommunications networks (internet) between a knowledge acquirer 46 and a knowledge provider 48. Dedicated software in the form of an App installed on the mobile device 44 may be used to affect this transfer. The App may be used to allow direct connection between remote knowledge acquirers and may include a directory for identifying and contacting suitable knowledge providers (closed group) as required. The App may also be used to schedule appointments with knowledge providers.

The App or the hat 10 may also capture other parameters for recording as collected data. Some of this data may be available to the knowledge provider during the session while other parameters are merely recorded for later analysis. The parameters may be environmental parameters, such as temperature, noise, vibrations or light conditions at the workplace, or biometric parameters of the knowledge acquirer, such as body temperature, heart rate, perspiration etc. Also, location systems such Global Positioning System may be incorporated as a parameter. The App may include a video/audio processing software package which can operate on standard operating systems (such as Windows, Android or Apple OS). The App and software at the knowledge provider's terminal or computing unit should have communications software that has high bandwidth capacity for conferencing between the knowledge acquirer with the mobile communications device and the knowledge provider's computer and their OS. A communications system should be provided at the knowledge provider's location to enable the encoding and decoding of the audio/video information.

As illustrated in Figure 3, a central switching computer 40 is used to identify on a computer network the location of the knowledge provider and the knowledge acquirer and provide appropriate switching instructions when instructed to do so. An artificial intelligence unit 50 may be used to capture two-way audio/visual information to record the interaction between the technician 46 and the knowledge provider 48 for future reference, audit and training purposes.

In addition to the switching computer 40, the artificial intelligence unit 50, or incorporated thereinto, an instruction module may also be provided. The instruction module may also be configured to dynamically study the workplace, ascertain the problem and give instructions. This may involve identifying a training module relevant to the situation and providing instructions from a pre-recorded collection of instructions, which may be based on previous interactions between knowledge provider and knowledge acquirer. The instruction module may be self-learning from previous interactions between knowledge provider and knowledge acquirer.

The hat 10 also includes a power source which is connected to the camera 14, the speakers 18, the controller 22, the communications module and the projector 21. The power source is preferably a rechargeable battery.

As illustrated in Figure 3, the hat 10 can include a display 26 disposed under the brim 29 of the hat or at the edge of a user's eyesight for displaying various information, such as for example connectivity status, whether a particular component is switched on and battery level. LED's may be provided for displaying this information.

Hat 10 relies on near synchronous transmission of video and audio between the two ends of the call, i.e. between the knowledge provider and the knowledge acquirer, to allow the passing of critical information and giving of directives in real time. Because of this requirement, it is important that the knowledge acquirer is aware that the link is in synchronisation or not. Due to the nature of the Telecommunications networks there maybe occasions when the video and audio arriving at the knowledge provider would be time lagging that of knowledge acquirer. This means the audio and visual instructions or directives will be late when relayed via the speakers and projector. Under these conditions the knowledge acquirer must be alerted to condition and stop any manual operation or actions until such time as the time lag has decreased to a predetermined time, which is preferably 0.5 Sec or less.

A synchronisation indicator or network status indicator 28 is provided to measure the delay in the round trip from the Hat 10 and back again which is preferably equal to or less than 0.5 Sec. A warning indicator shall be used by the knowledge acquirer to ascertain if the delay is satisfactory or not. It shall indicate the following states:

a) Green - Less than or equal to 0.5 Sec

b) Amber - Less than 1 Sec but greater than 0.5 Sec.

c) Red - Greater than 1 Sec.

To determine network status, the controller 22 is configured for measuring packets of information from both directions and gain information on the round-trip delay. Using the protocol for the streaming video and associated audio, a special ID marker is inserted into the data stream, preferably in the direction from the hat 10 to the knowledge provider. Upon receipt of the ID marker at the terminal of the knowledge provider, it is immediately sent back as an extra data packet. The ID marker includes a unique code and sequence stamp and does not interfere with the decoding of audio packets. The handling of missing or out of sequence packets shall be handled in the same way as the TCP/IP packet handler. The lag performance monitor shall use a smoothing algorithm to enable an average of the individual lags to be calculated. Considering that the lag values are expected to follow a standard statistic natural log distribution curve the following equation shall be used :

L = e ^a2

Where L = mode value of lag

m = mean of L

s = standard deviation (0.25)

The algorithm should take no more than lOOmSec to produce an initial value of L As the indicator 28 is incorporated into the underside of the left-hand side of the brim 29 of the hat 10, the knowledge acquirer can be warned without impeding or distracting them. The indicator 28 is preferably positioned to remain only just visible in upper corner of the wearers left eye.

During a call, the link shall be monitored for delay performance. If the monitoring algorithm indicates that the link has a delay of 0.5 Sec or less, then the indicator light should be set to give a "green" status on the Network status indicator. If at any time the performance unit starts to detect a delay of 0.5 to 1 sec the Network status indicator shall change to an "amber" colour. Further, if the delay that is detected is over 1 sec for a period 0.25secs or longer the Network status indicator shall change its colour to "red".

A system status indicator 30 is also provided. Once the controller 22 is satisfied with the internal check and the battery capacity level is more than 50% the status indicator 30 shall be turn on in a "green" state. If the battery capacity level is between 10 and 50% then the status indicator 30 shall be in an "amber" state. Once the battery capacity level goes below 10% the status indicator 30 shall be in a "red" state.

To enable a warning to the knowledge acquirer without impeding them or distracting them, the status indicator light 30 is preferably incorporated into the underside of the right-hand side of the brim 29 of the hat 10. It shall be positioned to remain only just visible in upper corner of the knowledge acquirer's eye. Another form indicator provides a light source to shine into the eye of the knowledge acquirer to get their attention may also be provided

The hat 10 is preferably configured to be as unobtrusive as possible so as to provide minimal interference to the work of the knowledge acquirer. This is important as in some situations, for example when the knowledge acquirer simultaneously needs to operate another device, such as a laptop, tablet or specialised equipment and displays, or electric control panels, for example. The hat 10 is an example of headwear which is part of a system for instructing or training a knowledge acquirer working at a workplace remote from a knowledge provider person. Such a system includes a remote terminal 52 or computing unit, which may be a personal computer configured for viewing the first video signal and for receiving input instructions from the knowledge provider. The input instructions are preferably made via a mouse or stylus type device and are preferably in the form of markings made on a screen displaying the first video signal. The markings may be in the form of predetermined icons, text or freehand lines. X-Y coordinates may be provided on the screen to assist with locating a particular point or tracking a length. A live stream of pixel by pixel information may be provided to create a live drawn picture. The input instructions are converted to a second electric signal and communicated to the headwear via the communications module for display at the worksite by the projector 21.

In use, the software running on the computing unit 52 may include a button for launching a visual assistance program, which displays a series of predetermined icons on the screen and which can be dragged onto the display to enable the knowledge provider to communicate information with the knowledge acquirer. Freestyle lines and/or predetermined shapes may also be used and projected onto the workplace via the projector. Furthermore, an image of the workplace may also be used and once projected onto the workplace, the knowledge acquirer will be easily able to identify inconsistencies and readily modify the workplace accordingly. When the visual assistance program is launched, different option screens can be either minimised or excluded from being displayed at the knowledge provider's computer. Once the visual assistance program is launched, the knowledge acquirer is notified that the projector is about to start, which preferably occurs after a predetermined delay, such as two seconds, to avoid the projector being pointed into a person's eyes.

Either the knowledge acquirer via the mobile device or the knowledge provider via the computing unit 52, may have access to controls of the projector, such as colour, contrast and brightness.

The computing unit 52 software shall obtain positional co-ordinates from computing unit's standard operating system mouse pointer position and send this data to the hat 10. Also, in this data is information as to the selection of a symbol over freehand and the type of symbol chosen (i.e. an arrow).

The controller 22 will receive the data from the knowledge provider's computing unit 52 and interpret the positional information into the correct position on the projected surface. In addition, the information data received will give a code for the symbol chosen by the knowledge provider and displayed on the surface of the object viewed.

Based on the knowledge provider's input, the computing unit 52 software shall interpret the code and construct a new scene for the projector which is a picture of the same symbol, with a black background projected in the same co-ordinates as the pointer on the computing unit 52. This symbol is now projected onto the surface that the knowledge acquirer is viewing.

The position of the symbol is added to the composite picture that is seen by the camera 14. The knowledge provider can now see their local mouse pointer projected at the knowledge acquirer's location. At this point there can be an option to remove the local pointer as the knowledge provider can control the remote symbol from their mouse and display in the knowledge provider's computing unit 52.

When the visual assistance program is no longer required, the knowledge provider can activate a button on the browser program to turn off the symbol projection. To carry this operation out, the knowledge provider will click the left-hand mouse button once to restore the local mouse pointer on the screen. The knowledge provider can then click on the deactivate button on the browser page. This will cause the program to send data to hat 10 to cease the projection of the symbol and turn off the projector.

Hat 10 may send data back to computing unit 52 to confirm the visual assistance program has been turned off. When this confirmation is received at the computing unit 52, the program will shut down the options window which is running in the background and reset the button on the display to the original activate button.

Instead of using symbols, or in addition to, a free hand image maybe projected, operating in much the same way as described above, though with the options window open this time the knowledge provider would select a "freehand" mode of operation for the visual assistance program. On selection of this option the option window shall be minimised so as not to block the video picture from hat 10.

An example use of the system is in the remote training of a technician in the telecommunications industry performing fibre splicing in an underground pit. The technician may have completed a theory of fibre splicing through an "on-line" registered training organisation, as well as all his basic Fibre cable and OH&S training. The technician has at hand all the necessary equipment to carry out fibre splicing (that is the heat fusing of two individual fibres together to make one continuous length of fibre) and has scheduled an appointment to complete his practical test in fibre splicing using a fibre splicing machine AST123.

Once the technician has arrived at the location for the test and set up the area around the underground pit for safe working, he will remove the pit lid and remove the fibre cable as identified. He will then prepare the two ends of the fibre cables for the splicing operation and set up a splicing table and the splicing machine on the same.

At the time of the appointment, the technician places the hat 10 on their head and powers it up by pushing a power button. During this time, a technician would typically check that the batteries are charged. The microphone, headphones, camera and projector may be powered through the controller 22. A communication link is then activated to pair the hat 10 with a mobile device 44. The App is then launched on the mobile device 44 for controlling the communication between the technician and a knowledge provider.

The App may perform a number of checks of hat 10 to ensure it is operational.

Once the technician is satisfied that the hat 10 is setup and operating correctly, they move to a position in front of the splicing machine where they can operate the machine with their hands. The technician may be seated in position with the mobile device 44 on their hip and can then check the video from the camera 14 and adjust the focus so the splicing machine is in clear view. At the prearranged time, the technician contacts the Trainer. This may be via a calendar entry with both parties contact details, or alternatively it can include selecting from a menu a conference link item and scrolling down a list of connection addresses until he sees the required address. Once selected, the App will automatically establish a connection via the Telecommunications or data communications network using a standard internet protocol for video/audio links to the knowledge providers remote computing unit 52.

The Trainer will receive an internet call through his client application which is part of the initialisation of the system. Once connected, the Trainer and technician will be able to talk to and listen to each other. At the same time the Trainer will be able to see the view from the technician's camera. Provided that sufficient bandwidth and capacity of the link are available, the Trainer should have a picture-perfect view of what the technician is looking at in real time. Once the task is complete, the call can be terminated through the App on the mobile device. If desirable, the hat 10 may be removed and replaced with a standard hard hat, while the hat 10 can be powered down and placed on charge for later use.

The system has a number of uses, in particular in the industrial sectors and telecommunications, and will have particular application in the area of training and assistance with troubleshooting. The following is a list of applications in no particular order:

• Teaching training practical skills

• Assessment of practical skills on-the-job

• Providing expert assistance to staff who are unclear what is required

• Auditing of work completed to the correct standard

· Verification of work completed in a safe and healthy manner.

The system could be extended to other industries that have staff or independent workers who require expert assistance in the field or remote location. Whilst the above description includes the preferred embodiments of the invention, it is to be understood that many variations, alterations, modifications and/or additions may be introduced into the constructions and arrangements of parts previously described without departing from the essential features or the spirit or ambit of the invention. Although the above embodiments have been described for use by a technician, it will be appreciated that the system may be used by many different people in many different situations.

Throughout this specification and the claims which follow, unless the context requires otherwise, the word "comprise", and variations such as "comprises" and "comprising", will be understood to imply the inclusion of a stated integer or step or group of integers or steps but not the exclusion of any other integer or step or group of integers or steps.

The reference in this specification to any prior publication (or information derived from it), or to any matter which is known, is not, and should not be taken as an acknowledgment or admission or any form of suggestion that that prior publication (or information derived from it) or known matter forms part of the common general knowledge in the field of endeavour to which this specification relates.