TECHNICAL FIELD

Embodiments herein relate to a method and a control node for controlling a mining rig in a mining environment. Furthermore, a computer program and a carrier are also provided herein.

BACKGROUND

Mining rigs are used to perform operations in a mining environment. The operations performed by the mining rigs may relate, for example, directly to drilling operations, or to supporting drilling operations in the mining environment. The mining rigs operating in the mining environment may be of many various types used for different purposes, such as:

- face drilling rigs and tunnelling rigs, e.g., for drilling blast holes to drive the face of the drift in a mine or a tunnel in a construction work site,

- rock reinforcement rigs for installing rock bolts and rock mesh,

- production drill rigs for drilling long holes for blasting or media holes,

- scaling rigs for removing loose rock from the rock surface, and

- shotcrete rigs for consolidating the rock surface by spraying a layer of concrete or resin onto the rock surface.

Independent of the type of mining rig used in the mining environment, in order to control the mining rig, the mining rig typically has a cabin for an operator to sit, which cabin comprises controls for the operator to use when operating the mining rig. However, for simplifying the process of controlling the mining rig, and for improving the safety of the operator of the mining rig, the mining rig may sometimes be controlled remotely. In these cases, as illustrated in Fig. 1 , the mining rig is equipped with cameras enabling the operator to sit at a remote operating station and see live video feeds from the cameras and remotely operate the mining rig using remote controls. This creates an operating environment similar to that of actually operating the mining rig from the cabin.

While controlling a mining rig remotely may be like controlling the mining rig as if sitting inside the cabin, differences and problems of how to perform mining operations remotely may still arise. To better learn how to operate mining rigs remotely, the operators may thus first be trained using a digital twin of the mining rig. The digital twin is a simulation of how the mining rig would operate in the mining environment based on inputs from the operator. In this way, the operator can be trained to be able to control the mining rig remotely for most operations. However, when the operator needs to perform certain complex mining operations involving many steps, such as rock reinforcement, remote operation of the mining rig may prove to be too difficult. Therefore, the operator may need to be present in the mining environment to be able to perform these certain complex mining operations. As an example, it may be difficult to execute a drill plan comprising a series of drilling operations at different locations, since the drill plan requires the operator to see if it is possible to drill a hole at an exact location according to the drill plan or if the location has to be changed slightly due to detailed characteristics of the rock surface. These slight details may be difficult to see when the operator is only provided with present technology. Furthermore, vision provided by cameras on mining rigs may in some cases be obstructed by various parts of the mining rig, and in these cases, the camera vision does not provide enough information for the operator to complete a certain operation remotely.

SUMMARY

An object of embodiments herein is to improve remote control of mining rigs performing operations in a mining environment. In particular, it is an object to achieve improvements to operational safety and reducing the need for physical inspections.

According to an aspect, the object is achieved by a method for controlling a mining rig in a mining environment. The method comprises obtaining sensor data from a plurality of sensors. The plurality of sensors comprises one or more first sensors for sensing the mining environment and one or more second sensors for sensing the mining rig in the mining environment. The method further comprises generating a virtual mining model based on the obtained sensor data. The virtual mining model comprises a combined three-dimensional real-time representation of the mining rig and the mining environment. The combined three-dimensional real-time representation of the mining rig and the mining environment comprises a coordinate system for establishing a positioning relationship of the mining rig relative to the mining environment in real time. The method additionally comprises triggering one or more operations to be performed by the mining rig in the mining environment based on the generated virtual mining model. Triggering one or more operations comprises triggering one or more real-time operations.

According to another aspect, the object is achieved by a control node configured to control a mining rig in a mining environment. The control node is further configured to obtain sensor data from a plurality of sensors. The plurality of sensors comprises one or more first sensors for sensing the mining environment and one or more second sensors for sensing the mining rig in the mining environment. The control node is further configured to generate a virtual mining model based on the obtained sensor data. The virtual mining model comprises a combined three-dimensional real-time representation of the mining rig and the mining environment. The combined three-dimensional real-time representation of the mining rig and the mining environment comprises a coordinate system for establishing a positioning relationship of the mining rig relative to the mining environment in real time. The control node is further configured to trigger one or more operations to be performed by the mining rig in the mining environment based on the generated virtual mining model. To trigger one or more operations comprises to trigger one or more real-time operations.

It is furthermore provided herein a computer program comprising instructions, which, when executed on at least one processor, cause the at least one processor to carry out the method above. It is additionally provided herein a carrier comprising the computer program. The carrier is one of an electronic signal, an optical signal, an electromagnetic signal, a magnetic signal, an electric signal, a radio signal, a microwave signal, or a computer-readable storage medium.

By enabling positioning relationships of the mining rig relative to the mining environment in a three-dimensional real-time representation, it is possible to trigger the one or more operations based on perspectives otherwise obscured or not easily observable by present technology. Since these new perspectives of the mining rig in relation to the mining environment are enabled to be observed in a real-time manner by the three-dimensional real-time representation, it is enabled to trigger of the one or more operations based on an improved contextual awareness which results in operational and safety improvements of remotely controlled mining operations. In particular, due to the three-dimensional real-time representation, it is no longer necessary to have an operator present in the mining environment to inspect the mining rig in relation to the mining environment. Instead, the one or more operations are enabled to be triggered and performed under remote control in real time.

In some embodiments herein, triggering the one or more operations may comprise triggering one or more real-time operations. When triggering the one or more real-time operations, operational and safety improvements of remotely controlled mining operations are further increased. This is since triggering the one or more real-time operations enables controlling the mining rig to quickly avoid potential accidents and/or potential performance degrading events occurring in real time. BRIEF DESCRIPTION OF THE DRAWINGS

Examples of embodiments herein are described in more detail with reference to attached drawings in which:

Fig. 1 is a schematic block diagram illustrating a remotely controlled mining rig according to prior art.

Fig. 2 is a schematic block diagram illustrating a scenario according to embodiments herein.

Fig. 3 is a flowchart depicting a method according to embodiments herein.

Fig. 4 is a schematic block diagram illustrating embodiments herein.

Figs. 5a-b are schematic block diagrams illustrating embodiments of a control node.

DETAILED DESCRIPTION

Fig. 2 is a schematic overview depicting a mining environment 100, a mining rig 20, a virtual mining model 20’, and an operator 60.

The mining environment 100 may typically be part of an underground mine and may comprise a series of tunnels and spaces covered by rock surface 27. The mining environment 100 may further comprise one or more infrastructure objects for supporting mining operations such as lighting elements, supporting beams, etc. In the mining environment 100, various mining operations may be performed, such as drilling and blasting holes, reinforcing tunnels, and extracting materials.

The mining rig 20 may be any mining rig suitable to perform any of the above- mentioned mining operations in the mining environment 100. For example, the mining rig 20 may be a drill rig, such as any of a face drill rig, a tunnelling rig, a rock reinforcement rig, a production drill rig, a scaling rig, or a shotcrete rig. The mining rig 20 may be mobile and may be capable of driving, operating, and moving within the mining environment 100. In addition to the mining rig 20 being capable of being remotely controlled, the mining rig 20 may comprise a cabin from which an operator may be able to control the mining rig 20. The mining rig 20 may alternatively be cabin-free, and only be capable of being remotely controlled. The mining rig 20 may comprise one or more booms 22, 23, for example, telescopic booms, controllable by the mining rig 20. The one or more equipment may comprise any suitable mining equipment needed to perform mining operations in the mining environment 100, such as any one or more of the above-mentioned mining operations. For example, the one or more equipment may be, e.g., a feeder for feeding a rock drill, a rock drill for drilling in the mining environment 100, a bolting equipment for reinforcing rock strata by installing rock bolts or cables in the mining environment 100, a scaling hammer to hammer loose rock, or a shotcrete rig consolidating the rock surface by spraying layers of concrete onto the rock surface. The equipment may be attached to any part of the mining rig such as attached to the one or more booms 22, 23.

The virtual mining model 20’ is a real-time representation of the mining rig 20 in the mining environment 100. The virtual mining model 20’ may be a digital twin of the mining rig 20 in the mining environment 100. The virtual mining model 20’ comprises a combined three-dimensional real-time representation 21. The combined three- dimensional real-time representation 21 of the mining rig 20 and the mining environment 100 comprises a coordinate system for establishing a positioning relationship of the mining rig 20 relative to the mining environment 100 in real time. In other words, the combined three-dimensional real-time representation 21 of the mining rig 20 and the mining environment 100 is used to keep track of a current location of the mining rig 20, relative to the mining environment 100 in real time. For example, the combined three- dimensional real-time representation 21 enables a way to represent the current state and location of the mining rig 20 and the mining environment 100 in three dimensions in real time. In this way it may be possible to observe the mining rig 20 and the mining environment 100 in angles and/or perspectives not possible by present technology. By using the combined three-dimensional real-time representation 21 of the mining rig 20 and the mining environment 100, it is possible to safely, at a remote location, perform complex operations otherwise not possible to perform remotely due to poor or otherwise obstructed vision. The combined three-dimensional real-time representation 21 of the mining rig 20 and the mining environment 100 is generated based on sensor data from a plurality of sensors. The plurality of sensors comprises one or more first sensors 24a-c for sensing the mining environment, and one or more second sensors 25a-c for sensing the mining rig in the mining environment. One or more sensors out of the plurality of sensors may be attached to the mining rig 20, or be part of an infrastructure in the mining environment 100. A wide range of configurations for how to position the plurality of sensors are possible. Generally, the sensors are placed such that the sensors, at least in collaborations, may be able to have a complete or near-complete view surrounding the mining rig 20, e.g. no, or few parts should obfuscate the combined view of the sensors. For example, any one or more sensors out of the plurality of sensors may respectively be attached to any one or more out of: a roof of the mining rig 20, the one or more booms 22, 23, a cabin of the mining rig 20 such as on top of the roof of the cabin, in a rear-position of the mining rig 20, in a front-position of the mining rig 20, in the highest most position of the mining rig 20 e.g., sensing at least partially downwards, in the lowest position of the and/or mining rig e.g., sensing at least partially upwards. Furthermore, one or more sensors out of the plurality of sensors may be arranged at various locations in the mining environment 100 and/or attached to other vehicles operating in the mining environment 100. These sensors may independently or in collaboration pre-scan an area of the mining environment 100 in advance of the rest of sensors out of the plurality of sensors. The mining rig 20 may in these scenarios obtain other sensor data to position the mining rig 200 in the pre-scanned area of the mining environment 100.

The combined three-dimensional real-time representation 21 of the mining rig 20 and the mining environment 100 may be subject to one or more real-time conditions. For example, the combined three-dimensional real-time representation 21 may not represent a state of the mining rig 20 and/or the mining environment 100 older than a certain period of time, e.g., at least some of the sensor data used for generating the virtual mining model 20’ may not be older than 100 milliseconds (ms). The real-time condition may for example pertain to sensor data used for monitoring real-time movements of the mining rig 20.

The operator 60 may remotely operate the mining rig 20 in the mining environment 100 observing the combined three-dimensional real-time representation 21 by, for example, using at least one display 61 , using Augmented Reality (AR) or Virtual Reality (VR) headset or glasses 62 or any other suitable tool for AR or VR. The operator 60 may, using the combined three-dimensional real-time representation 21 , observe, from any perspective, how and where the mining rig 20 is positioned in the mining environment 100. The operator 60 may further be able to observe operations performed by the mining rig 20 in the mining environment 100. In this way, the operator 60 may be able to observe the mining rig 20 and/or the mining environment 100 from angles and locations which would have otherwise been difficult or impossible to observe, e.g., due to being obscured by parts of the mining rig 20 or by parts of, and/or infrastructure in, the mining environment 100. For example, the operator 60 may freely move around a visual representation of the combined three-dimensional real-time representation 21 and observe the mining rig 20 and/or the mining environment 100 from any suitable perspective, even from normally impossible perspective such as a perspective from inside the rock foundation of the mining environment 100. The operator 60 may then provide one or more control inputs for controlling the mining rig 20 e.g., in real time, and continue to observe the mining rig 20 and/or the mining environment from any suitable perspective while operating the mining rig 20 e.g., in real time.

Real time as used herein may indicate one or more real-time conditions to be fulfilled. In general, any real-time condition mentioned herein may relate to ensuring a quick enough update such that it is possible for the operator 60 to observe changes to the mining rig 20 and/or the mining environment 100 in the combined three-dimensional realtime representation 21, and by remote control, have time to control the mining rig 20 to avoid an unwanted situation, e.g. to avoid accidents. Real-time conditions may depend on a speed of the mining rig 20, and/or computational speed relating to performing the embodiments herein. Real-time conditions may relate to soft real-time conditions and/or hard real-time conditions. Real-time conditions herein may be a soft or hard real-time conditions. The hard real-time conditions may relate to ensuring that an operation is performed within a hard time limit, e.g. within 80ms or 100ms, wherein it may further be verified that the hard time limit cannot be missed. For hard real-time conditions, it is not acceptable that the hard time limit is missed as the hard real-time conditions may relate to safety considerations. To further ensure safety, a missed hard time limit may trigger an emergency halt of operations in the mining environment 100. The soft real-time conditions may relate to attempting to perform an operation within a soft time limit for ensuring a high performance. For soft real-time conditions, performance degradation may be associated with missing the soft time limit.

The one or more real-time conditions may relate to some specific sensor data, e.g., from one or more sensors out of the plurality of sensors or may relate to sensor data obtained from one or more sensors out of the plurality of sensors which senses specific parts of the mining rig 20 and/or the mining environment 100. In one example, sensor data used for monitoring real-time movements of the mining rig 20 may not be older than a certain period of time, e.g., 100ms. In some embodiments, the one or more real-time conditions may relate to that real-time operations to be performed by the mining rig 20 may need to be performed within a certain period of time, e.g. not more than 100ms after being triggered. The one or more real-time conditions in embodiments herein may be used to ensure safety of operating the mining rig 20 in the mining environment 100. For example, if any of the one or more real-time conditions is not fulfilled, a halt of the mining rig 20 may be triggered and/or an alert or an alarm may be triggered for indicating to the operator 60 that a real-time condition has not been fulfilled. Methods herein may be performed by a control node 40 or any other suitable device, e.g., the mining rig 20 or a suitable drill rig. In some embodiments the control node 40 is part of the mining rig 20. In some other embodiments the control node 40 may be a remote node in communication with the mining rig 20, for example, in a control room remotely located from the mining rig 20.

Embodiments herein relate to combining sensor data from the plurality of sensors and apply the sensor data to a virtual mining model 20’ of the mining rig 20 and the mining environment 100. Using the virtual mining model 20’, a three-dimensional VR or AR representation may be generated which may augment to the operator 60, visual information needed to complete a complex mining operation in real-time such as an entire rock bolting or drilling operation from a remote location. The presented visual information further makes new functionalities possible to be performed remotely with greater efficiency, such as improved collision avoidance. This is since it is, due to the embodiments herein, possible, e.g., for the operator 60, to observe in three dimensions with depth-perception, the mining environment 100 and the mining rig 20 operating in the mining environment 100. Since the visual capabilities are improved, one or more operations, e.g. real-time operations, may be triggered based on the combined three- dimensional real-time representation 21. Using the embodiments herein the mining rig 20 may further be taught how to perform a series of mining operations, for example a drill plan, and thus increase the efficiency of mining operations in the mining environment 100. These operations may be performed without any real-time conditions and hence may be planned in advance. In this way, it is possible to advantageously utilize the combined three-dimensional real-time representation 21 to in advance plan the one or more operations to be performed by the mining rig 20. The one or more operations may first be performed virtually using the combined three-dimensional real-time representation 21, recorded, and then performed by the mining rig 20. For example, this may be performed by planning the series of mining operations by first inspecting the locations in the mining environment 100 of each of the mining operations to be performed in various angles in the coordinate system of the combined three-dimensional real-time representation 21. In this way, it is possible to determine how and where to perform each of the respective mining operations. The series of mining operations may then be triggered and performed automatically with minimal or no supervision and/or further control by the operator 60.

A number of embodiments will now be described, some of which may be seen as alternatives, while some may be used in combination. Fig. 3 shows example embodiments of a method for controlling the mining rig 20 in the mining environment 100. In some embodiments, the method may be performed by the control node 40, or any suitable device such as the mining rig 20. The method comprises the following actions, which actions may be taken in any suitable order. Optional actions are referred to as dashed boxes in Fig. 3.

Action 301. The method may comprise obtaining a three-dimensional representation of the mining rig 20. In some of these embodiments, the obtained three- dimensional representation of the mining rig 20 is a Computer-Aided Design (CAD) drawing of the mining rig 20.

Action 302. The method comprises obtaining sensor the data from the plurality of sensors. The plurality of sensors comprises the one or more first sensors 24a-c for sensing the mining environment 100 and the one or more second sensors 25a-c for sensing the mining rig 20 in the mining environment 100. Any one or more of the sensors may be attached to, or part of the mining rig 20. Alternatively, one or more of the sensors may be attached to the rock surface 27 or infrastructure of the mining environment 100.

The sensor data may be obtained from each of the respective sensor out of the plurality of sensors based on a respective sensing frequency, i.e., refresh rates, of each respective sensor. For example, some sensor data may be obtained at a low frequency, and some sensor data may be obtained at a high frequency, i.e., more often.

In some embodiments, the sensor data of different sensors may be obtained in batches, i.e., where several types of sensor data are obtained at the same time. In some embodiments, the sensor data are pre-processed before being obtained. For example, the mining rig 20 may be associated with a computing unit, e.g. attached to, coupled with, or connected to the mining rig 20, which first pre-processes the sensor data. Pre-processing the sensor data may involve filtering out anomalies of the sensor data, and/or to prepare the sensor data for transmission. For example, in some scenarios, networking capacity, e.g. bandwidth, may be limited. Hence, in these scenarios it may be necessary to perform computations locally on the mining rig 20, e.g. to aggregate data, and/or to filter out some data to reduce data sent over the network.

Obtaining the sensor data may involve receiving the sensor data from the plurality of sensors. Thus, the sensor data may be transmitted from the plurality of sensors, e.g. orchestrated by a communication unit which is connected to, coupled with, or attached to, the mining rig 20. Obtaining the sensor data may thus further involve receiving the sensor data at the control node 40. In some embodiments, the plurality of sensors comprises at least one high resolution sensor and at least one low resolution sensor. The at least one high resolution sensor captures high resolution sensor data of the mining environment 100. In a corresponding manner, the at least one low resolution sensor may capture low resolution data for establishing real-time position and/or status of the mining rig 20 and the mining environment 100. In some of these embodiments, the high resolution sensor data captured by the at least one high resolution sensor is of a higher resolution than the low resolution sensor data captured by the low resolution sensor. High resolution sensor data may comprise more than a first quantity of sensor data points. Additionally or alternatively, high resolution sensor data may comprise sensor data with accuracy and/or precision higher than a second threshold. Low resolution sensor data may comprise less than a third quantity of sensor data points. Additionally or alternatively, low resolution sensor data may comprise sensor data with accuracy and/or precision lower than a fourth threshold. The at least one low resolution sensor may capture sensor data at a higher rate than the at least one high resolution sensor. For example, the low resolution sensor data may be captured at e.g. 20Hz, and the high resolution sensor data which may be captured at a much lower frequency e.g. once per day or every 15 ^th minute depending on technology used and the demand for high resolution sensor data. In these embodiments, the high resolution sensor data may only be captured rarely and provide a basis for detailed data of the mining rig 20 and/or the mining environment 100. The low resolution data may instead be captured often, and while not as detailed as the high resolution sensor data, the low resolution sensor data may provide a basis for establishing real-time positions and status of the mining rig 20 and/or the mining environment 100.

The at least one high resolution sensor may comprise any one or more out of:

- a rotating laser scanner,

- a plurality of two-dimensional cameras,

- a time-of-flight camera,

- a solid state Light Detection and Ranging, Lidar, sensor, and

- a structured light three-dimensional scanner.

The rotating laser scanner may be a Light Detection and Ranging (Lidar) sensor, for example, for scanning the mining environment 100. The rotating laser scanner may scan the mining environment 100 and provide information of the features of the mining environment 100, for example, represented by a set of positions of scanned parts of the mining environment 100 also referred to as a point cloud. Images captured by the plurality of two-dimensional cameras may be derived to form a high resolution three-dimensional representation of the mining rig 20 and mining environment 100 by using photogrammetry.

The time-of-flight camera may be a camera which captures one or more images or video and may further estimates, for every pixel in the one or more images or video, a distance from the mining rig 20 to different positions in the mining environment 100. The time-of-flight camera may provide two-dimensional and/or three-dimensional information for each pixel of the one or more images or video captured. The time-of-flight is a low-cost camera typically suitable for dark areas and may work over long distances. Furthermore, the time-of-flight camera may provide distance information at high accuracy and/or precision.

The solid state Lidar may be a Lidar embedded in a micro-chip. The solid-state Lidar may further scan the mining environment 100 without any moving parts.

The structured light three-dimensional scanner is which can scan a recognizable pattern. Based on the recognizable pattern, e.g. a striped pattern, the scanner may estimate the distance and three-dimensional space between the scanner and the recognizable pattern. This is since the scanner may know and/or may be able to derive how the pattern looks at different distances and/or angles.

Action 303. The method further comprises generating the virtual mining model 20’ based on the obtained sensor data. The virtual mining model 20’ comprises the combined three-dimensional real-time representation 21 of the mining rig 20 and the mining environment 100. The combined three-dimensional real-time representation 21 of the mining rig 20 and the mining environment 100 comprises the coordinate system for establishing the positioning relationship of the mining rig 20 relative to the mining environment 100 in real time. In this way, the virtual mining model 20’ provides a digital twin which represents the mining rig 20 in relation to the mining environment 100. This enables observing the mining rig 20 and the mining environment 100 from any perspective in real- time. Further, when controlling the mining rig 20, e.g. remotely, it is thus possible to see from any angle and perspective, how the mining rig 20 operates within the mining environment 100.

In some embodiments, generating the virtual mining model 20’ comprises generating the coordinate system comprised in the combined three-dimensional real-time representation 21 of the mining rig 20 and the mining environment 100. Generating the coordinate system may comprise obtaining one or more relative distances between one or more of the plurality of sensors and a reference point of the mining rig 20. Furthermore, generating the coordinate system may further comprise estimating one or more distances from the mining rig 20 and the mining environment 100. The one or more distances from the mining rig 20 and the mining environment 100 may be one or more distances from the mining rig 20 and different parts of the rock surface 27. The estimation of the one or more distances may be based on the obtained sensor data and the one or more relative distances between the one or more of the plurality of sensors and the reference point of the mining rig 20. Generating the coordinate system may further comprise determining one or more positions associated with the mining rig 20 and the mining environment 100. The one or more positions may be determined relative to the reference point of the mining rig 20. Determining the one or more positions associated with the mining rig 20 may be based on the one or more distances from the mining rig and the mining environment 100. To estimate the one or more distance from the mining rig 20 and the mining environment 100, the sensor data obtained by the plurality of sensors may be compared with the reference point of the mining rig 20. In this way, it is possible to determine where the mining rig 20 parts are located, or where the mining environment rock surface 27 are located, etc. In some embodiments, the plurality of sensor may comprise many different sensors, and thus by using all of the sensors, it may be possible to estimate the one or more distances from the mining rig 20 and the mining environment 100 with high precision and/or accuracy. Determining one or more positions associated with the mining rig 20 and the mining environment 100 may further comprise identifying one or more positions of parts of any one or more out of the one or more booms 22, 23. The one or more positions of the parts of any one or more out of the one or more booms 22, 23 may further be used to determine how the one or more booms 22, 23 may move, or may be able to move, in relation to the mining rig 20 in the mining environment 100. Identifying the one or more positions of the specific parts of any one or more out of the one or more booms 22, 23 may be based on one or more sensors out of the plurality of sensors, e.g. in particular sensors sensing at least part of the one or more booms 22, 23 such as any one or more out of length sensors, angle sensors, scanners, and cameras.

Determining one or more positions associated with the mining rig 20 and the mining environment 100 may further comprise identifying one or more static locations in the mining environment 100. The one or more static locations may further be used as a reference point for determining the position of the mining rig 20 position in relation to the mining environment 100. The number of sensors used, and/or the number of types of sensors used, may in some scenarios relate to the increase in accuracy and/or precision of the estimated one or more distances from the mining rig 20 and the mining environment 100. This is since by using sensor data of many different sensors, it may be possible to verify first sensor data from a first sensor and/or to detect anomalies in the first sensor data by comparing the first sensor data from with other sensor data of other sensors which senses the same or related objects as the first sensor. As an example, sensor data obtained from a Lidar sensor may indicate the positions of one or more parts of the one or more booms 22, 23. In this scenario, the sensor data obtained from the Lidar sensor may be verified based on sensor data obtained from one or more sensors sensing the state, e.g. angles, of one or more joints in the one or more booms 22, 23.

The relative distances between the plurality of sensors and the reference point of the mining rig 20 may in some scenarios at least partially be derived based on sensor data from one or more sensors attached to the mining rig 20 with a known position on the mining rig 20, e.g. relative the reference point of the mining rig 20. For example, one or more angle sensors and/or one or more extension sensors may measure the extension and/or angles the one or more beams. In this way, it may be possible to determine distances and positions of at least parts of the one or more respective beams in relation to the mining rig 20 based on the sensor data of the one or more angle sensors and/or the one or more extension sensors.

Generating the virtual mining model 20’ may further comprise deriving information of any one or both of the mining rig 20 and the mining environment 100 based on a combination of sensor data obtained from at least two differing sensors in the plurality of sensors. This may also be referred to as sensor fusion.

For example, the plurality of sensors may comprise a set of sensors which senses the current state of the mining rig 20, for example if a particular part of the mining rig 20 is extended or not and by which angle. Another set of sensors of the plurality of sensors may be used for scanning the front-view of the mining rig 20, and yet another set of sensors of the plurality of sensors may be used for scanning the rear-view of the mining rig 20. While none of the above-mentioned set of sensors may on their own accurately describe the mining rig 20 and the mining environment, the combination of the sensor data of the above-mentioned sets of sensors may be used to derive information of the position and state of both the mining rig 20 and the mining environment 100. This information may further be used to generate the virtual mining model 20’. Furthermore, in some scenarios the same part of the mining environment 100 may be scanned and/or sensed by several different sensors, e.g. of same or of differing sensor type.

When the plurality of sensors comprises the at least one high resolution sensor, and when the at least one high resolution sensor comprises the plurality of two-dimensional cameras, generating the virtual mining model 20’ may further comprise deriving the combined three-dimensional real-time representation 21 of the mining rig 20 and the mining environment 100, based on a plurality of two-dimensional images captured by the plurality of two-dimensional cameras. In other words, the combined three-dimensional real-time representation 21 of the mining rig 20 and the mining environment 100 may be generated based on photogrammetry of two-dimensional images of the mining rig 20 and the mining environment 100.

In some embodiments, generating the virtual mining model 20’ is further based on the obtained three-dimensional representation of the mining rig 20, e.g. as obtained in action 301. In this way, it is possible to more accurately represent the mining rig 20 in the combined three-dimensional real-time representation 21 of the mining rig 20 and the mining environment 100.

In some embodiments, generating the virtual mining model 20’ is further based on a planned model of the mining environment 100. The planned model may be obtained from a suitable server, database, or a memory. The planned model may comprise information of how the mining environment 100 is planned to be after one or more mining operations, e.g. which resulting physical attributes and/or spatial arrangements the mining environment 100 will have after the one or more mining operations. For example, the planned model may comprise a bolt pattern to be installed in the rock surface 27 of the mining environment 100. Furthermore, the planned model may comprise a tunnel to be drilled in the in the rock surface 27 of the mining environment 100. In this way, the resulting virtual mining model 20’ may comprise a combination of the sensed mining environment 100 based on the obtained sensor data, and a planned model of the mining environment 100 based on the obtained planned model.

Action 304. As a part of generating the virtual mining model 20’ e.g. as in Action 303, the method may comprise producing a visual representation of the virtual mining model 20’. The visual representation may comprise a viewing position and a viewing orientation in the coordinate system which will further be exemplified in Fig. 4. In these embodiments, the viewing position and viewing orientation in the coordinate system corresponds to a location and orientation in the mining environment 100. In other words, the visual representation may be a snapshot of the current state of the mining rig 20 and the mining environment as seen from a location in the mining environment 100 which corresponds to the viewing position and viewing orientation in the coordinate system. Producing the visual representation of the virtual mining model 20’ may fulfil the one or more real-time conditions. In some embodiments, the one or more real-time conditions may further relate to that the visual representation is not representing a state of the mining rig 20 older than a certain time period. For example, the visual representation of the virtual mining model 20’ is updated at a frequency high enough, e.g. any interval between 10Hz, 60Hz, or 120Hz, to ensure safe remote operation of the mining rig 20 in relation to operational speed of parts of the mining rig 20, time for signaling and signal processing, time for generating the virtual mining model 20’, and time to visualize the virtual mining model such that the operator 60 is enables and/or has time to trigger the mining rig 20 to perform time-critical operations.

In some embodiments, the visual representation may further illustrate parts of the mining rig 20 and/or the mining environment as at least partially transparent. In this way, it may be possible for the operator to see through these parts of the mining rig 20 and/or the mining environment 100.

Action 305. In some embodiments, the operator 60 may observe the visual representation of the virtual mining model 20’. In response to an operator 60 observing the visual representation of the virtual mining model 20’, the method may comprise obtaining one or more control inputs from the operator 60. The one or more control inputs may be used to control the mining rig 20 and/or to control what to observe in the visual representation of the virtual mining model 20’.

The one or more control inputs may be obtained from the operator 60 by means of the operator 60 inputting the one or more control inputs by using a remote operator station comprising a joystick and a plurality of buttons. The remote operator station may be any workstation suitable for the operator to use for inputting the one or more control inputs, e.g., a computer, a laptop, mobile phone, tablet, or similar. The virtual representation of the virtual mining model 20’ may be presented to the operator 60 on the at least one display 61.

The one or more control inputs may also be obtained from the operator 60 by means of the operator 60 inputting the one or more control inputs by using a VR and/or AR environment. The virtual representation of the virtual mining model 20’ may be presented to the operator 60 in the VR and/or AR environment, e.g. by using VR headset or glasses 62. The operator may in some of these embodiments operate the mining rig by physical or virtual operator controllers or a combination of both. A wide range of controllers are therefore applicable, for example: VR gloves, dedicated VR controllers, game station controllers, a tele remote operating panel. In some embodiments it may be possible to provide the one or more control inputs by means of gestures in the VR/AR environment. For example, the operator 60 may point at a position in the mining environment 100 wherein a certain mining operation shall be performed.

The one or more obtained control inputs from the operator 60 may be associated with the operator 60 performing one or more virtual operations in the virtual mining model 20’. The one or more virtual operations may correspond to the one or more operations to be performed by the mining rig 20 in the mining environment 100. In other words, the operator 60 may perform the virtual operations and control the mining rig 20 as observed from the visual representation of the virtual mining model 20’. These virtual operations may then translate to one or more operations triggered to be performed by the mining rig 20 in the mining environment 100. This may be a seamless experience such that the operator 60 is in this way experiencing controlling the mining rig 20 in the mining environment 100 e.g., in real time. The operator 60 may also use the virtual mining model 20’ to plan and to perform a series of operations. The operator 60 may then input one or more control inputs to trigger the mining rig 20 in the mining environment 100 to perform a series of mining operations, for example a drill plan. The series of mining operations may then be performed automatically without further input from the operator 60.

Action 306. The method comprises triggering one or more operations to be performed by the mining rig 20 in the mining environment 100 based on the generated virtual mining model 20’. The one or more operations may then be performed based on vision not otherwise provided for by present technology. Triggering the one or more operations to be performed by the mining rig 20 may further be based on the one or more control inputs obtained from the operator 60, e.g., as in Action 305.

The one or more operations triggered to be performed by the mining rig 20 in the mining environment 100 may comprise one or more real-time operations. In other words, the real-time operations may be triggered based on a real-time condition. For example, the one or more operations need to be performed, or started to be performed, within a set amount of time, e.g. 100ms. When the one or more operations are real-time operations, it is possible to remotely control the mining rig 20 in a safer and/or more efficient manner as it is possible to react, within a short amount of time, to potential dangers and/or other performance degrading events that may occur in the mining environment 100. For example, it may be observed in the combined three-dimensional real-time representation 21, that there are obstacles, e.g. people, walls, and/or vehicles, in the way of the mining rig 20 when driving forward or operating equipment of the mining rig 20 in a certain direction. In these scenarios, when the one or more operations are real-time operations, it is thus possible to trigger an operation to stop, or an operation to change direction in a minimal time.

In embodiments where the mining rig 20 comprises the one or more booms 22, 23, triggering the one or more operations to be performed by the mining rig 20 may comprise operating each of the one or more booms 22, 23 in the mining environment 100, e.g., in real time, respectively. For example, the mining rig 20 may, based on the one or more operations, be triggered to operate a first boom of the one or more booms 22, 23. This may be to activate a drill attached to the first boom. In this way, using the drill of the first boom, the mining rig 20 may then commence a drilling operation in the mining environment 100.

Action 307. Based on the one or more control inputs, the method may comprise triggering an update of the visual representation of the virtual mining model 20’ such that the updated visual representation comprises any one or both of an updated viewing position and an updated viewing orientation in the coordinate system. In other words, the operator may use control inputs to observe the mining rig 20 and the mining environment in the virtual mining model 20’ from any suitable perspective. The operator 60 may move around freely in the combined three-dimensional real-time representation 21 of the mining rig 20 and the mining environment 100.

The above embodiments will now be further explained and exemplified below. The embodiments below may be combined with any suitable embodiment above.

Fig. 4 illustrate an example scenario for observing the virtual mining model 20’ in different perspectives. Fig. 4 illustrates the combined three-dimensional real-time representation 21 of the mining rig 20 and the mining environment 100 and the corresponding produced visual representation. In this scenario, the visual representation is observed by the operator 60 on the at least one display 61 , in a VR environment, or in an AR environment in the AR/VR headset or glasses 62. The combined three-dimensional real-time representation 21 of the mining rig 20 and the mining environment 100 comprises two different viewing positions, a first viewing position 51 , and a second viewing position 52 in the coordinate system. The first viewing position 51 correspond to a first location in the mining environment 100 and the second viewing position 52 correspond to a second viewing position in the mining environment 100. Both of the first and second viewing positions 51, 52 have respective viewing orientations in the coordinate system which correspond to respective viewing orientation in the mining environment 100.

The visual representation may be produced based on the first viewing position 51 and its respective orientation. This visual representation may then visualize the mining rig 20 and the mining environment 100 for the operator 60 in real-time, as if the operator 60 was observing the mining rig 20 and mining environment 100 from an initial location and orientation in the mining environment 100. In these embodiments, the initial location and orientation in the mining environment 100 may correspond to the first viewing position 51 and the viewing orientation of the first viewing position 51. In some embodiments, the visual information provided to the operator 60 does not comprise the necessary information for the operator 60 to be able to safely and/or efficiently perform the one or more operations. This may be since the visual presentation may now show the operator 60, the mining rig 20 from the perspective necessary to determine how to perform the one or more operations. The operator 60 may then input one or more control inputs for moving the first viewing position 51 to a second viewing position 52 in the coordinate system, e.g. as in action 307 above. The visual representation may then be updated to visualize the mining rig 20 and the mining environment 100 for the operator 60 in realtime, as if the operator 60 was observing the mining rig 20 and mining environment 100 from an updated location and orientation in the mining environment 100. In these embodiments, the updated location and orientation may correspond to the second viewing position 52 and the viewing orientation of the second viewing position 52. The operator 60 may, using the perspective of the updated visual representation, have visual information of high enough quality to perform the one or more operations. While illustrated herein as two different viewing positions 51 , 52, the visual representation may change and be updated to be produced based on any number of different viewing position in a seamless real-time manner, e.g. updating several times a second. The produced visual representation may appear to the operator 60 as a video representation of the mining rig 20 and the mining environment 100, wherein the operator 60 may choose from which viewing position and which orientation the mining rig 20 and the mining environment should be observed from.

Sensor and sensor data details. The at least one high resolution sensor in some embodiments herein may be a high resolution scanner used to capture detailed data of the mining environment 100 surrounding the mining rig 20. However, the high resolution scanner may generally be too slow and require too much computer power to capture live motion in real time and may therefore mostly be used for scanning static parts of the mining environment 100. The high resolution scanner may be a rotating scanner which scans the environment 360 degrees around the mining rig 20. The sensor data obtained by the high-resolution scanner may comprise information about the mining environment 100 which may be used to derive a precise and accurate positioning of the mining rig 20 and the one or more equipment which may be attached to the mining rig 20. In some embodiments, the high resolution scanner may be a laser scanner, but may also be any other type of suitable scanner, such as a Radio Detection and Ranging (Radar) sensor.

The at least one low resolution sensor in some embodiments herein may be a fast scanner used to capture real time movements, i.e. dynamics, of the mining rig 20 and equipment attached to the mining rig 20 such as the one or more booms 22, 23. The fast scanner may scan the one or more booms 22, 23 when for example the operator positions and moves the respective one or more booms 22, 23. The fast scanner will further be able to capture real time movements using less bandwidth than the high resolution scanner as it may capture the mining rig 20 and the mining environment 100 using low resolution. The fast scanner may further detect if there are changes to the mining environment, e.g. if the surrounding rock from the high resolution scan has changed in any major way, e.g. more than a threshold. If so, a new high resolution scan may be needed and may in some embodiments be triggered to be performed at least at the area of change. In some embodiments, a camera may be used instead of, or in combination with the fast scanner to capture real time movements. In some embodiments, the fast scanner may be a laser scanner, but may also be any other type of suitable scanner, such as a Radar sensor.

In some embodiments herein, the one or more first sensors 24a-c out of the plurality of sensors may comprise any one of the following cameras for sensing, i.e. capturing images or video of, the mining environment 100:

- one or more Red Green Blue (RGB) cameras which may provide one or more colored images of the mining environment 100 at low cost,

- one or more Infrared (IR) cameras which may provide good vision in areas with dust. The one or more IR cameras may provide additional thermal information which may provide information for detecting human presence in the mining environment 100,

- one or more stereo cameras which may provide three-dimensional images and may provide distance information with high precision and/or accuracy for long ranges.

The one or more second sensors 25a-c of the plurality of sensors may be used to capture the mining rig 20 real-time movements in the mining environment 100. For example, these sensors may capture movements of the one or more equipment and/or the one or more booms 22, 23, which may be attached to the mining rig 20 in various embodiments herein. The sensor data obtained from these sensors may be used to reposition the mining rig 20, and/or the parts of the mining rig 20 in the combined three- dimensional real-time representation 21 of the mining rig 20 and the mining environment 100. In some embodiments, the one or more second sensors 25a-c out of the plurality of sensors may additionally or alternatively comprise any one or more out of the following sensors for sensing the mining rig 20 in the mining environment 100:

- one or more angle sensors for sensing which angle the mining rig 20 is extending any one or more of the attached one or more equipment such as the one or more booms 22, 23,

- one or more extension sensors for sensing how degree of extension of the attached one or more equipment such as the one or more booms 22, 23,

- one or more rotation sensors for sensing a rotation degree of which the mining rig 20 has rotated any one or more of the attached one or more equipment such as the one or more booms 22, 23,

- one or more actuators at pivot points, e.g. attached to a respective part out of one or more parts of the one or more booms 22, 23, for sensing a respective tilt angle of the one or more parts of the one or more booms 22, 23,

- one or more calibration sensors for calibrating sensors and/or movements of the one or more booms 22, 23 attached to the mining rig 20, e.g. telescopic booms may be bended downwards due to their own weight and gravity and may need calibration to ensure their current position and/or current state.

In some embodiments, using any one or more out of the one or more angle sensors, the one or more extension sensors, and the one or more rotation sensors, a position and movement of joints in each respective boom of the one or more booms 22, 23 of the mining rig 20 may be calculated and/or controlled.

To perform the method actions above, e.g. actions 301-307, the control node 40 is configured to control the mining rig 20 in the mining environment 100. The method actions above may also be performed by the mining rig 20 or any suitable drill rig capable of performing the action methods above. The control node may be located at a remotely located control room or at the mining rig 20. The control node 40 may comprise an arrangement depicted in Figs. 5a and 5b. The control node 40 may comprise an input and output interface 500 e.g. for communicating with the mining rig 20 and/or the plurality of sensors. The input and output interface 500 may also visualise the visual representation to the operator 60, and further provide the means for the operator 60 to enter the one or more control inputs. The input and output interface 500 may comprise a receiver (not shown) and a transmitter (not shown), both which may be wired or wireless.

The control node 40 may further be configured to, e.g. by means of an obtaining unit 501 in the the control node 40, obtain a three-dimensional representation of the mining rig 20. In some of these embodiments, the obtained three-dimensional representation of the mining rig 20 is a CAD drawing of the mining rig 20.

The control node 40 may further be configured to, e.g. by means of the obtaining unit 501 in the the control node 40, obtain sensor data from a plurality of sensors. The plurality of sensors comprises one or more first sensors 24a-c for sensing the mining environment 100 and one or more second sensors 25a-c for sensing the mining rig in the mining environment 100.

The control node 40 may further be configured to, e.g. by means of a generating unit 502 in the the control node 40, based on the obtained sensor data, generate a virtual mining model 20’. The virtual mining model 20’ comprises a combined three-dimensional real-time representation 21 of the mining rig 20 and the mining environment 100. The combined three-dimensional real-time representation 21 of the mining rig 20 and the mining environment 100 comprises a coordinate system for establishing the positioning relationship of the mining rig 20 relative to the mining environment 100 in real time.

In some embodiments, the control node 40 may further be configured to, e.g. by means of the generating unit 502 in the the control node 40, generate the virtual mining model 20’ further based on the three-dimensional representation of the mining rig 20.

The control node 40 may further be configured to, e.g. by means of the generating unit 502 in the the control node 40, generate the virtual mining model 20’ by deriving information of any one or both of the mining rig 20 and the mining environment 100 based on a combination of sensor data obtained from at least two differing sensors in the plurality of sensors.

The control node 40 may further be configured to, e.g. by means of the generating unit 502 in the the control node 40, generate the virtual mining model 20’. In these embodiments, generating the virtual mining model 20’ comprises generating the coordinate system comprised in the combined three-dimensional real-time representation 21 of the mining rig 20 and the mining environment 100 by:

- obtaining one or more relative distances between one or more of the plurality of sensors and a reference point of the mining rig 20, - based on the obtained sensor data and based on the one or more relative distances between the one or more of the plurality of sensors and the reference point of the mining rig 20, estimating one or more distances from the mining rig 20 and the mining environment 100, and

- based on the one or more distances from the mining rig and the mining environment 100, determining one or more positions associated with the mining rig 20 and the mining environment 100, wherein the one or more positions are determined relative the reference point of the mining rig 20.

In some embodiments, the plurality of sensors comprises at least one high resolution sensor and at least one low resolution sensor, wherein the at least one high resolution sensor captures high resolution sensor data of the mining environment 100, and wherein the at least one low resolution sensor captures low resolution data for monitoring real-time movements of the mining rig 20.

In some embodiments, the high resolution sensor data captured by the at least one high resolution sensor is of a higher resolution than the low resolution sensor data captured by the low resolution sensor, and wherein the at least one low resolution sensor captures sensor data at a higher rate than the at least one high resolution sensor.

For example, the high resolution sensor data may be sensed at a higher frequency than low resolution sensor data. Additionally or alternatively, the high resolution sensor data may comprise sensor data sensed by one or more accurate and/or precise sensors than the one or more sensors sensing the low resolution sensor data. Additionally or alternatively, the high resolution sensor data may comprise images and/or videos comprising more pixels than low resolution sensor data. Additionally or alternatively, the high resolution sensor data may comprise more accurate and/or precise distance measuring than the low resolution sensor data.

In some embodiments, the at least one high resolution sensor comprises any one or more out of:

- a rotating laser scanner,

- a plurality of two-dimensional cameras

- a time-of-flight camera,

- a solid state Lidar sensor, and

- a structured light three-dimensional scanner.

In some embodiments, the at least one high resolution sensor comprises the plurality of two-dimensional cameras. In some of these embodiments, the control node 40 may further be configured to, e.g. by means of the generating unit 502 in the the control node 40, generate the virtual mining model 20’ further comprises deriving the combined three-dimensional real-time representation 21 of the mining rig 20 and the mining environment 100, based on a plurality of two-dimensional images captured by the plurality of two-dimensional cameras.

The control node 40 may further be configured to, e.g. by means of a producing unit 503 in the the control node 40, produce a visual representation of the virtual mining model 20’, wherein the visual representation comprises a viewing position 51, 52 and a viewing orientation in the coordinate system. The viewing position 51 , 52 and viewing orientation in the coordinate system may correspond to a location and orientation in the mining environment 100. In these embodiments, producing the visual representation of the virtual mining model 20’ may fulfil a real-time condition.

The control node 40 may further be configured to, e.g. by means of a triggering unit 504 in the the control node 40, trigger one or more operations to be performed by the mining rig 20 in the mining environment 100 based on the generated virtual mining model 20’.

The control node 40 may further be configured to, e.g. by means of the obtaining unit 501 in the the control node 40, in response to an operator 60 observing the visual representation of the virtual mining model 20’, obtain one or more control inputs from the operator 60.

In some embodiments, the one or more control inputs is obtained from the operator 60 by the operator 60 inputting the one or more control inputs by using a remote operator station comprising a joystick and a plurality of buttons. In these embodiments the virtual representation of the virtual mining model 20’ is presented to the operator 60 on at least one display.

In some embodiments, the one or more control inputs is obtained from the operator 60 by the operator 60 inputting the one or more control inputs by using a VR environment, in these embodiments, the virtual representation of the virtual mining model 20’ is presented to the operator 60 in the VR environment.

In some embodiments, the one or more control inputs is obtained from the operator 60 by the operator 60 inputting the one or more control inputs by using an AR environment. In these embodiments, the virtual representation of the virtual mining model 20’ is presented to the operator 60 in the AR environment.

The one or more obtained control inputs from the operator 60 may further be associated with the operator 60 performing one or more virtual operations in the virtual mining model 20’. In these embodiments, the one or more virtual operations may correspond to the one or more operations to be performed by the mining rig 20 in the mining environment 100.

The control node 40 may further be configured to, e.g. by means of the triggering unit 504 in the the control node 40, trigger the one or more operations to be performed by the mining rig 20 further based on the one or more control inputs obtained from the operator 60.

In some embodiments, the mining rig 20 comprises a boom. In these embodiments, the control node 40 may further be configured to, e.g. by means of the triggering unit 504 in the the control node 40, trigger the one or more operations to be performed by the mining rig 20 to operate the boom 21 in the mining environment 100 e.g., in real time.

The control node 40 may further be configured to, e.g. by means of the triggering unit 504 in the the control node 40, based on the one or more control inputs, trigger an update of the visual representation of the virtual mining model 20’ such that the updated visual representation comprises any one or both of an updated viewing position and an updated viewing orientation in the coordinate system.

The embodiments herein may be implemented through a respective processor or one or more processors, such as the processor 560 of a processing circuitry in the control node 40 depicted in Fig. 5a, together with respective computer program code for performing the functions and actions of the embodiments herein. The program code mentioned above may also be provided as a computer program product, for instance in the form of a data carrier carrying computer program code for performing the embodiments herein when being loaded into the control node 40. One such carrier may be in the form of a CD ROM disc. It is however feasible with other data carriers such as a memory stick. The computer program code may furthermore be provided as pure program code on a server and downloaded to the control node 40.

The control node 40 may further comprise a memory 570 comprising one or more memory units. The memory 570 comprises instructions executable by the processor in the control node 40. The memory 570 is arranged to be used to store e.g. information, indications, data, configurations, sensor data, three-dimensional representations, and applications to perform the methods herein when being executed in the control node 40.

In some embodiments, a computer program 580 comprises instructions, which when executed by the respective at least one processor 560, cause the at least one processor of the control node 40 to perform the actions above.

In some embodiments, a respective carrier 590 comprises the respective computer program 580, wherein the carrier 590 is one of an electronic signal, an optical signal, an electromagnetic signal, a magnetic signal, an electric signal, a radio signal, a microwave signal, or a computer-readable storage medium.

Those skilled in the art will appreciate that the units in the control node 40 described above may refer to a combination of analog and digital circuits, and/or one or more processors configured with software and/or firmware, e.g. stored in the control node 40, that when executed by the respective one or more processors such as the processors described above. One or more of these processors, as well as the other digital hardware, may be included in a single Application-Specific Integrated Circuitry (ASIC), or several processors and various digital hardware may be distributed among several separate components, whether individually packaged or assembled into a system-on-a-chip (SoC).

When using the word "comprise" or “comprising” it shall be interpreted as nonlimiting, i.e. meaning "consist at least of".

It will be appreciated that the foregoing description and the accompanying drawings represent non-limiting examples of the methods and apparatus taught herein. As such, the apparatus and techniques taught herein are not limited by the foregoing description and accompanying drawings. Instead, the embodiments herein are limited only by the following claims and their legal equivalents.