REMOTE ASSISTANCE SYSTEM AND APPARATUS

The present invention relates to a remote assistance system and apparatus .

The present invention relates particularly, although by no means exclusively, to a remote assistance system and apparatus for a mine .

The present invention also relates to a mobile site apparatus for use as a part of a remote assistance system.

The present invention relates particularly, although by no means exclusively, to a mobile site apparatus for use as a part of a remote assistance system for a mine .

The present invention also relates to a method of operating a mine that comprises using a remote assistance system at the mine .

The present invention also relates to a method of operating a mine that comprises using a mobile site apparatus at the mine.

The present invention relates particularly, but not exclusively, to a remote assistance system and apparatus and a mobile site apparatus as a part of that system for carrying out maintenance of plant and equipment at a mine and the following description focuses on this application to a certain extent. It is emphasised that this is not the only possible application for the remote assistance system and apparatus of the present invention and the mobile site apparatus of the present invention. A related company of the applicant operates a number of iron ore mines in remote locations around the world. Each of these mines includes substantial plant and equipment , such as excavators , haulage trucks , crushers , and conveyors to mine and process mined ore at the mine site prior to transporting the mined ore by train to shipping ports . The plant and equipment requires non- scheduled maintenance as a consequence of breakdowns . Currently, the staff at each mine includes skilled engineers and skilled technicians who carry out maintenance work on the plant and equipment. Down time of plant and equipment, for whatever reason, is a significant issue given the considerable pressure on mines to produce mined ore and the substantial loss of revenue as a result of any downtime. Employing, transporting and housing skilled engineers to work at remote mines are a significant challenge .

The present invention is a remote assistance system and apparatus to facilitate the provision of technical assistance to personnel at mines and a mobile site apparatus that can be used as a part of the system by the personnel at the mine which enables them to be instructed by a skilled person located remotely of the mine site in the provision of maintenance and other technical tasks . This has the potential to enable more efficient maintenance of plant and equipment at mines than is possible now. As indicated above, the present invention , and hence the benefits of the invention described in this paragraph, are not confined to mining applications and extend to a number of other industries, for example where monitoring and maintaining plant and equipment are required on site and involve skilled input.

The remote assistance system and apparatus and the mobile site apparatus of the present invention can also be used for other applications at mines . The applications include mine planning applications.

The remote assistance system comprises setting up and operating a support centre that services a number of mines and is located away from the mines and staffing the support centre with skilled operators who work with operators at the mines , with each mine site operator having a mobile site apparatus that enables the operator to be in contact with the support centre.

The support centre is set up with support stations in communication with mobile support apparatus via a communications network, such as a Local Area Network (LAN) and/or Wide Area Network (WAN) . The support station and the mobile support apparatus facilitate communications between the operators thereof so that the operator of the support station can assist the operator of the mobile support apparatus to perform maintenance or other tasks at a site remote from the support station.

The support centre is set up so that the operators have access to other operators at the centre (and therefore can take advantage of the skills and experience of these operators) , a large number of sources of technical information (for example computerised databases and direct telephone or other connections to equipment suppliers) . Each mobile site operator is equipped with (a) audio (b) visual, (c) data collaboration and (d) data display capabilities. For example, a mobile mine operator may be equipped with a headset that incorporates audio capabilities . By way of further example, the data display capability may be a screen positioned on the end of an arm extending from the headset at a location to one side of the eyes of the mobile mine operator. The system includes a capability to transfer audio, visual and "projected data", such as image and video data, between the mobile site operator and the operator at the support centre .

In use, when a need for maintenance is identified, for example at a crusher at a mine site, a mobile mine operator carries out an initial diagnostic evaluation of the problem at the mine site and then, as required, physically inspects and carries out work on the crusher. A data transfer link is established between the mobile site operator and an operator at a support station at the support centre, whereby the support station operator is able to visually inspect the crusher with the mobile site operator via video and image data supplied from the mobile site apparatus. The audio link enables direct communication between the operators as the inspection continues . Depending on the circumstances , the support station operator is able to access information from sources of information available to the operator and to provide technical information to the mobile site operator. By way of example, the support station operator may directly contact equipment suppliers for real-time information . Depending on the circumstances , the support station operator can project (i.e. communicate) data, such as wiring diagrams , from the support desk to the mobile site apparatus so that the mobile site operator can view the data on the screen and carry out maintenance/repair operations on-site.

In the context of the mining application of the present invention, according to the present invention there is provided a remote assistance system for providing technical assistance to mines that comprises a support centre, one of more mines requiring technical assistance from time to time, a communications network link between operators in the support centre and mobile mine operators at sites at the mines at which technical assistance is required, and capabilities to provide the support centre operators with audio and visual information from the mines via the mobile mine operators at the mine sites, and capabilities to provide the mobile mine operators with data that can be visually displayed to the mobile mine operators and other information aurally to facilitate technical assistance at the mine sites.

The advantages of the present invention include :

• Consolidating knowledge in a single location. The knowledge base at the single location may extend across multiple technical disciplines and provide an opportunity for a higher knowledge base from which to solve problems at a mine .

• Locating the support centre in a population centre where there is not the same difficulty employing and retaining specialist support as is the case with mines in remote locations .

The remote assistance system may be configured so that a support centre operator is the driver of the entire system and as such has access to all the software functionality of the system. In this situation, the mobile mine operators are, in effect, mobile extensions of the capabilities of the support centre operator . In this situation, the computer processing capacity of system apparatus carried by a mobile mine operator may be minimal .

The remote assistance system may comprise bidirectional voice communication, video communication, and data (information) collaboration between an operator at the support centre and a mobile mine operator.

Information collaboration may be achieved by setting up a "remote desktop" type connection. This means that in any given situation a support centre operator and a mobile mine operator are able to connect to the same common virtual workspace . This workspace may be used to share documents , drawings and any other information required by the operator at the mine .

The remote assistance system may comprise unidirectional information transfer from a support centre operator to a mobile mine operator.

The unidirectional information transfer may comprise a support centre operator selecting a window of a data and image display screen at the support centre and transferring this window to a display screen carried by a mobile mine operator.

The support centre may be configured to store voice, data and image signals received from or transmitted to the mobile site apparatus against time, date, location and/or session information. This feature facilitates subsequent analysis of the stored information .

The support centre may be configured to store voice, data and image signals received from or transmitted to the mobile site apparatus against time, date, location and/or session information to thereby facilitate selective retrieval and display at the support centre at least the data and/or image signals during subsequent transmission of signals to and receipt of signals from the mobile site apparatus. This feature makes it possible to return to a previous session, for example a session relating to a particular piece of equipment, that could not be completed during the previous session, for example because it was necessary to source a replacement part or access external sources for advice. This feature makes it possible for the support centre operator to quickly review the history of the previous session. This is a particularly useful feature where different operators are involved in the multiple sessions for the particular action.

According to the present invention there is also provided a support centre configured for communicating with a mobile site apparatus worn by an operator located remotely from the support centre, the support centre being configured for transmitting over a communications network at least voice and data signals to and for receiving over the communications network at least voice, data and image signals from the mobile site apparatus, the support centre being configured to store voice, data and image signals received from or transmitted to the mobile site apparatus against time, date, location and/or session information.

The support centre may be configured to store voice, data and image signals received from or transmitted to the mobile site apparatus against time, date, location and/or session information to thereby facilitate selective retrieval and display at the support centre at least the data and/or image signals during subsequent transmission of signals to and receipt of signals from the mobile site apparatus .

The support centre may be configured so that the support centre operator is the driver for transmitting and receiving at least the data and/or image signals .

According to the present invention there is also provided a mobile site apparatus worn by an operator located remotely from a support centre, the mobile site apparatus being configured for communicating with the support centre, the mobile site apparatus being configured for transmitting over a communications network voice, data and image signals to the support centre, and the mobile site apparatus being configured for receiving over the communications network voice , data and image signals to the support centre .

In the wider context of the present invention beyond the mining industry, according to the present invention there is provided a remote assistance system for providing technical assistance that comprises a support centre, one or more sites requiring technical assistance from time to time, a communications network link between operators in the support centre and mobile site operators at sites at which technical assistance is required, and capabilities to provide the support centre operators with audio and visual information from the sites via the mobile site operators at the sites, and capabilities to provide the mobile site operators with data that can be visually displayed to the mobile site operators and other information aurally to facilitate technical assistance at the sites .

The remote assistance system described in the preceding paragraph may comprise the other features of the above-described remote assistance system for providing technical assistance to mines .

According to the present invention there is also provided a mobile site apparatus for a mobile site operator comprising the following components :

(a) a headset;

(b) a processing unit, for example in the form of a wearable computer;

(c) a hand-held camera unit;

(d) a display unit; and (e) a battery pack.

The apparatus may comprise a backpack that supports at least the processing unit.

The display unit may be connected to the backpack, for example chest straps of the backpack, via a telescopic arm that is hinged at both the backpack and the camera unit to enable translational and rotational movement of the display unit.

The apparatus may comprise an additional fixed camera, which preferably forms part of the headset.

The apparatus may also comprise a laser pointer.

There may be one laser pointer associated with each camera . Each laser pointer may be directed so that the relevant camera will capture a laser dot created when a laser beam from the laser pointer illuminates a physical surface . This enables the mobile site operator to be able to readily indicate areas of interest to the support desk operator though the laser dot and ensures that the support desk operator is viewing the area of interest.

The apparatus may comprise a GPS , which is preferably attached to or incorporated in the processing unit and forms part of the backpack. Through transmission of GPS data the support desk operator is able to identify where the mobile site operator is working, and allows such data to be superimposed on a map viewable by the support desk operator.

According to the present invention there is also provided a method of operating a mine that comprises using the above-described remote assistance system at the mine. According to the present invention there is also provided a method of operating a mine that comprises using the above-described mobile site apparatus at the mine.

According to the present invention there is also provided a system for providing technical assistance from an operator of a support station to an operator of a mobile support apparatus located remotely of the support station; the system comprising: the support station and the mobile support apparatus , the support station and the mobile support apparatus being in communication via a communication network; the mobile support apparatus comprising wireless communications apparatus for wireless communication of at least voice and image data between the mobile support apparatus and the communications network; the mobile support apparatus further comprising at least one processor for processing at least voice and image data received from the support station and for displaying at least the image data on a display of the mobile support apparatus and for playing at least the voice data through speakers operatively associated with the mobile support apparatus; the mobile support apparatus further being adapted to capture voice data from a microphone operatively associated with the mobile support apparatus and to capture still and/or video image data from at least one camera operatively associated with the mobile support apparatus and to transmit the voice, still and/or video image data to the support station; and the support station being adapted to receive and play the voice data and to receive and display the still and/or image data and to selectively store in response to operator input at least the still and/or video images against any one or more of time , date , location and session data for subsequent retrieval and display on a display operatively associated with the support station.

The at least one camera may be operatively associated with a light source to indicate to the operator of the mobile support apparatus an area of a surface onto which said camera is focused. The light source may be a laser or alternatively a focused light beam.

The camera may be mounted to a head set of the mobile apparatus whereby an operator of the support station views an image consisting of at least part of a current field of view of an the operator of the mobile support apparatus .

The camera may be mounted in or onto a head set of the mobile support apparatus which plays voice data transmitted from the support station .

The mobile support apparatus may further comprise a hand held camera incorporating a light source, whereby an operator of the mobile support apparatus directs the light source to illiminate a location on a surface for which still and/or video data is to be captured by said camera for transmission to the technical assistance station .

The support station may be adapted to simultaneously display and store still and/or video transmitted by the mobile support apparatus.

A display that may be associated with the mobile support apparatus may be a passive display adapted to only receive data from the support station .

The support station may further comprise an image selection window for selecting an area of a computer desktop on which the support station is operating and transmitting the image displayed at the selected area to a display of the mobile assistance apparatus .

The selection and transmission of image data from the desktop on which the support station is operating may enable the mobile support apparatus to operate substantially independently of process intensive software applications, such as Computer Aided Drawing (CAD) software packages.

The display of the mobile support apparatus may be adapted for the operator of the mobile support apparatus to zoom in and out of the image data received and to pan and scroll the image data received.

The present invention is described further with reference to the accompanying drawings , of which :

Figure 1 is a diagram that shows the high level connectivity of an embodiment of a remote assistance system of the present invention;

Figures 2-14 are perspective views of an embodiment of a mobile site apparatus (and the separate components of the apparatus) that can be used a s a part of the remote assistance system shown in Figure 1 ;

Figure 15 is a diagram that illustrates the interconnection of the components of the mobile site apparatus shown in Figures 2-14;

Figure 16 is a diagram that illustrates the WLAN and networking configuration of the remote assistance system shown in Figure 1; Figure 17 is a diagram that illustrates the software communication for the remote assistance system shown in Figure 1 ;

Figure 18 is a diagram that illustrates the media site implementation for the remote assistance system shown in Figure 1 ;

Figures 19-21 are a series of screen shots of a display screen of a support centre operator.

The following description of the embodiments of the remote assistance system and apparatus and the mobile site apparatus shown in the drawings is separated into sections to make the description clearer.

1.1 Overview of the remote assistance system

With reference to Figure 1 , the embodiment of the remote assistance system shown in the drawings comprises (a) an apparatus carried by a mobile mine operator at a mine (the apparatus is referred to hereinafter as "the mobile site apparatus" and labelled as "MATE Site Kit" in the Figures) and (b) a support station (hereinafter referred to as "support desk") manned by one or more skilled operators, for example located in a support centre servicing a number of mines at a location remote from the mines, for example in a major population centre several thousand kilometres away from the mine .

It is noted that the remote assistance system may be used in relation to a plurality of mines connected to a single support centre .

It is also noted that one support desk may be connected at different times to a plurality of mobile mine operators , each carrying a mobile site apparatus . With this arrangement, the single support desk can support multiple remote locations at different times via individual sessions with the mobile mine operators .

Communication between a mobile site apparatus and the support desk is via a local area network (LAN) and a wide area network (WAN) .

The remote assistance system has the following functionality between the mobile site apparatus and the support desk:

• Bi-directional voice communication between the mobile mine operator and an operator manning the support desk .

• Video communication between the operators .

• Data (information) collaboration between the operators.

• Mobility (wireless access) for the mobile site apparatus .

Voice and video communication is achieved using a real-time streaming protocol over a User Datagram Protocol (UDP) stream. Typically, video is only transmitted from the mobile site apparatus back to the support desk .

Information collaboration in the remote assistance system is achieved by setting up a "remote desktop" type connection. This means that both sides can connect to the same common virtual workspace . This workspace is used to share documents , drawings and any other information required by the operator at the mine site .

Alternatively, information collaboration may comprise unidirectional transfer of information from the support desk operator to the mobile mine operator . This may be achieved by the support centre operator selecting a window of a data and image display screen at the support desk and transferring this window to a display screen carried by a mobile mine operator .

1.2 Hardware Design (Mobile Site Apparatus)

With reference to Figures 2-14, one embodiment of the mobile site apparatus comprises a number of discrete components , including:

• A headset.

• A processing unit in the form of a wearable computer.

• A hand-held camera unit.

• A display unit.

• A battery pack.

These components are packaged in lightweight, rugged casings that are worn in a back-pack style .

All major components are housed in anodised aluminium cases with a final system rating of IP67.

The block diagram shown in Figure 15 illustrates the main components of the system and the component interconnections .

1.2.1 Headset

The following components are integrated into the headset: • Peltor SportTac noise-reducing headphones .

• Microsoft LifeCam NX-6000 video camera.

• A laser pointer.

• Speakers and a microphone .

For safety the laser pointer is designed to be activated from the headset only.

The microphone flap can flip back up to the rest position to protect the camera lens (if the system is not in use) and to provide privacy for the headset wearer.

1.2.2 Processing Unit

The core of the processing unit is an iBase IB882 single-board computer (SBC) with the following specification:

• Intel Atom Z500 processor running at 533MHz;

• DDR2 RAM with IGB capacity;

• 8GB Compact Flash Memory;

• Dual Intel Gigabit LAN adapters ;

The processing unit includes an input/output board that accommodates :

• Two SparkLAN WMIR-215GN 802. Hn wireless IAN cards.

• A ϋ-Blox NEO-4S GPS module. The SBC has USB, PS/2, LAN and VGA interfaces for diagnosing issues with the processing unit that cannot be fixed remotely .

The processing unit has status LEDs that display:

• Power state (on/off) .

• Hard-drive usage .

• WLAN usage .

The PCBs are thermally coupled to the inner casing of the processing unit to dissipate heat generated inside the enclosure.

The PCBs are shock mounted to the internal surface of the enclosure to protect the electronics from any jarring that it may receive.

1.2.3 Hand-held Camera Unit

The hand-held camera unit has the following components :

• Logitech Quickcam Pro 9000 camera.

• A laser pointer.

• An LED array.

The camera provides higher resolution video than the headset camera and as such the unit is useful for more detailed inspection . The laser pointer and the LED array are designed to be activated by the operator only, via respective on/off buttons located on the rear of the camera unit.

The LEDs are provided to illuminate objects up to half a meter away in line with the optimal focus of the camera .

A flap is used to cover the LEDs and camera lens when the camera unit is not in use.

1.2.4 Display Unit

The display unit is an Apple iPod Touch 8GB that is housed in an anodised aluminium case. The case is attached to the chest straps of the backpack via a telescopic arm that is hinged at both the chest strap and the device end to enable allow for translational and rotational movement .

The casing allows access to the power button (top left hand side of the case) to turn on/off the device.

A protective film is placed on top of the screen that allow the operator uninhibited access the touch sensitive screen.

The iPod connects via USB cable to the processing unit, to allow the display to charge. It does not use this cable for connectivity.

The display unit connects wirelessly to one of the wireless cards on the processing unit for data collaboration . 1.2.5 Battery Pack

The battery pack is made up of 8 Lithium-ion rechargeable battery packs that will realistically charge to approximately lβVolts. The battery pack is capable of powering the mobile site apparatus for approximately 8 hours under normal loading conditions . The battery pack is charged via an external laptop-style power brick. Charge status LEDs are located on the top of the battery pack to display remaining battery charge levels. The battery pack has a screw on connector and cable to supply power to the processing unit. A simple rail mounted system is used to locate the battery pack on the frame and a screw on the bottom secures it in place .

1.3 Operating System

The mobile site apparatus runs a Microsoft Windows Embedded operating system (0/S) . This has a number of advantages including:

• Reduced footprint.

• Support for read-only file systems .

• Full Win32 application compatibility .

• Support for .Net framework .

• Support for Windows Server Update Services.

The reduced footprint is required for the 8GB storage drives used.

The drive has limitations on the number of read/write cycles it can go through. Therefore, limits should be placed on the amount of I/O to the drive while the mobile site apparatus is in operation . To facilitate such limits, the 0/S has the capability to lock the file system and create a virtual space in the RAM where all I/O to the disk is performed. This means that once the file system is locked (at the initial device setup) any changes made in software cannot be saved, unless the write filter is explicitly switched off by an IT administrator/Engineer that is competent with the process .

1.3.1 Wireless LAN

The system has two separate but identical wireless adapters. For security and compliance, the cards fulfil some basic requirements , namely :

• Support for 802. Ix.

• Protected Extensible Authentication Protocol .

• WPA with TKIP.

• Cisco Compatible Extensions version 4 or higher.

One adapter is used to connect to the wireless infrastructure at a mine and is used as a primary mode of communication. The card supports the 802. Hn standard and therefore makes use of 3 quarter wave dipole 2.4GHz mini whip antennas .

The other adapter is used to relay information from the display unit to the LAN. This communication has significantly lower power requirements . Therefore , a cable antenna is located on the inner casing of the processing unit, which is connected to this adapter. A GPS antenna module is similarly located on the inner wall of the processing unit, close to the mini whip antennas .

With reference to Figure 16, the following configuration is used:

• DHCP addressing for primary WLAN adapter.

• Static addressing for secondary adapter and display unit.

• Network Address Translation from WLAN 1 to WLAN 2.

• Access through site firewall (Ports: 23150 - 23157).

1.4 Software Architecture (End-to-End Application)

The remote assistance system is designed such that the operator at the support desk is the driver of the entire system and as such has access to all the software functionality of the system. The mobile mine operators are, in effect, mobile extensions of the capabilities of the support centre operator.

The software architecture conforms to a server/client type model , with a database incorporated for archiving purposes .

The three main components of the software model are:

• Server - located on the apparatus .

• Front End - located on the support desk workstation. • Database - SQL backend server located somewhere on the network of the applicant.

The model shown in Figure 17 illustrates the relationship between these components and the role of the support desk operator in the remote assistance system.

1.4.1 Session Implementation

The software design includes sessions to enable archiving of specific data captured in a particular interaction .

Before commencing a call to a mobile site apparatus , the support desk operator creates a new session, using information such as the date, a short description of the issue reported, etc.

The session is then used to store data such as photographs taken, video recorded and any notes taken during the call .

All session data is uploaded to the database as it is incurred. This feature allows efficient data analysis across the whole mine network which may facilitate more efficient mine operation.

In addition, data for any stored session can be retrieved and displayed as required at the support desk . This feature makes it possible to return to a previous session, for example a session relating to a particular piece of equipment, that could not be completed during the previous session, for example because it was necessary to source a replacement part or access external sources for advice. This feature makes it possible for the support desk operator to quickly review the history of the previous session . This is a particularly useful feature where different operators are involved in the multiple sessions for the particular action.

1.4.2 Contact List Implementation

In order to simplify the connection process , there is a contact list system at the software layer.

The contact list displays the status of a particular mobile site apparatus. The following table illustrates the possible states of a mobile site apparatus .

Table 1 Possible states of a mobile site apparatus

1.4.3 Audio/Video Implementation

Figure 18 illustrates the implementation of the media suite.

The media server is built into the server and similarly, the media client is built into the front end. Both ends have similar high-level implementations that comprise:

• An encoder .

• A decoder.

• Transmitter/receiver .

The encoders and decoders communicate directly with the audio/video hardware (microphone, speakers, etc), while the transmitter/receiver communicates via RTP over UDP/IP.

The encoder and decoder has been developed using the proprietary library StreamCoders MediaSuite.Net. They encode video to H.263/H.264 and audio to AAC/MPEG-4.

1.4.4 GPS Tracking Implementation

GPS data is output from the unit via a virtual serial port. The unit outputs standard NMEA GPS strings, which are interpreted by the server application, on the site apparatus , continuously .

When the support desk operator connects to the mobile site apparatus, the support operator requests the GPS data, which is transmitted as a string of data that includes : • UTC time .

• GPS coordinate .

• Ground Speed.

• Height .

• Satellites in view & their signal strengths.

This data is then displayed on the support desk operator' s screen along with a map location (such as Google Maps) - see following description of a display screen in the context of Figures 19-21.

1.4.5 Data Collaboration Implementation

The virtual workspace consists of a windows XP standard operating environment, which is located on a virtual server, that can be accessed anywhere on the network .

In situations where there is a plurality of mobile site apparatus at a plurality of mines, each display unit on the mobile site apparatus connects to a unique virtual machine, thus avoiding any conflict. The display unit has a VNC type client installed, with preconfigured settings - including workstation addresses, usernames etc .

The mobile site apparatus operator has "readonly" access to the machine, thus leaving the support engineer in charge of navigation of pages . The mobile site apparatus operator only has the ability to pan and zoom, when viewing documents. The support desk operator has a VNC client installed on his/her workstation . The operator can use the client to connect directly to the virtual machine . Once logged in the operator is able to navigate to the required documents/schematics/system and display the information across the screen for both parties to view .

This collaboration can be selective in the sense that the operator can select windows of information displayed on his/her screen at the support desk and transfer these screens to the display of the mobile site operator .

Once the collaboration is complete , the mobile site apparatus operator disconnects his/her session while the support desk operator logs out of the workspace, thus terminating the session.

1.5 Display Screen

Figures 19, 20 and 21 are snap-shots of one example of a support desk operator's screen.

The Figure 19 screen snap-shot is a screen that an operator sees during the course of a session. The screen includes windows that provide video images and still images from the handheld and headset cameras at particular times in a session. The still images provide the support desk operator with a snapshot of the progress of a session. The screen also includes windows for (a) session "notes" to be made by the operator, (b) GPS information, and (c) session collaboration.

The Figure 20 screen snap-shot is a session start-up screen that enables a support desk operator to commence a session and commence information storage and facilitate session retrieval . The Figure 21 screen snap-shot is a screen that provides basic information for a particular session

The above-described embodiment of the present invention is an effective and efficient system and apparatus for providing high level remote support to a mine site. In particular, the remote site apparatus is robust and convenient to carry.

Many modifications may be made to the embodiments of the remote assistance system and apparatus described above without departing from the spirit and scope of the invention .

By way of example, whilst the description and drawings of the embodiments refer to particular hardware and software, the present invention is not so limited and extends to any suitable hardware and software for carrying out the stated functions.