SENSORS

DESCRIPTION OF BACKGROUND

This disclosure relates to mining machines. Particularly this disclosure relates to support structure for measurements instruments providing information for visual driving assistant systems and autonomous operation arrangements of a mining machine.

Mining machines are often big machines operating in tight mining tunnels or galleries. These machines have a wide variety of different tasks including, for example, quarrying, drilling and transportation. They are commonly driven by a person. The driver may be located in a cabin of the mining machine or use a remote control for controlling the mining machine. Using a remote control is particularly beneficial when the visibility from the cabin is limited.

A commonly used solution is to provide one or more cameras to provide driving assistants like in passenger cars. These cameras are may be ordinary two- dimensional camera units, stereo camera pairs, thermal cameras. In addition to cameras other sensing devices, such as laser scanners, radars and similar may be used in collecting information. These sensors are typically located so that they are less prone to damage. This may be achieved, for example, by providing a protective cover or placing the camera to a cavity or similar protective form. However, many times a safe location is not optimal for collecting information. This is particularly a problem when the bucket, or other tool installed in front of the mining machine, is moved so that it will limit the sensors visibility. SUMMARY

In the following disclosure a mining machine with a bucket and boom comprises one or more measurement sensors that are arranged into a support structure that provides a possibility to see over the bucket is disclosed. This provides possibility to see the contents of the bucket and also the area in the front of bucket. This provides better working conditions and also increases safety when persons and other objects in front of the mining machine can be detected.

In an embodiment a mining machine is disclosed. The mining machine comprises a body comprising a first body portion and a second body portion, wherein the mining machine further comprises: a boom and a bucket arrangement attached to the first body portion; a measurement sensor; and at least one support structure attached to the first body portion, wherein the at least one support structure is configured to support the measurement sensor, wherein the at least one support structure is arranged to extend above an upper surface of the first body portion so that the measurement sensor is able to see behind the bucket when the bucket is in a lowered position.

It is beneficial to have measurement sensors in a support structure that provides a possibility to equip the mining machine with measurement sensors that able to see to the bucket and also behind the bucket. This helps driver and/or autonomous systems to operate safely and also to measure the contents of the bucket. The elevated location provides a possibility to incline the measurement sensors in a manner that allows measuring over the bucket when the bucket is in a lowered position. The elevation required to see over the bucket depends on the bucket and how high the bucket is raised.

In an implementation the support structure is an arm extending from the upper surface of the first body portion. It is beneficial that the support structure extends from the upper surface of the first body portion so that location of the support structures stay same in relation to the bucket and the measurement sensors are forward directed.

In an implementation the mining machine comprises at least two arm support structures, wherein an arm is located on both sides of the boom. It is beneficial to use two separate arms so that the movement of the boom is not limited.

In an implementation the support structure is an arch extending above the boom. It is beneficial to use a arch as it is a stable and can be shaped and positioned so that it doesn't limit the movement of the boom .

In an implementation the support structure is movable and comprises position sensors for determining the position of the support structure. It is beneficial to have a movable support structure so that measurement sensors can be brought to a useful location at all times.

In an implementation the support structure is configured to extend and retract. It is beneficial to have a support structure to extend and retract so that the height of the support structure is variable and the measurement sensors can be raised and lowered when needed.

In an implementation the mining machine further comprises a processing circuitry configured to compute the location of measurement sensors based on the determined position of the support structure. It is beneficial to be able to measure and use the location of measurement sensors as the information may be used in constructing three-dimensional and other models and it is beneficial to use a plurality of sensors in the construction.

In an implementation the at least one support structure comprises a cover configured to protect measurement sensors. It is beneficial to use additional covers to protect measurement sensors in mine because in the mining environment measurement sensors could broken by falling rock and other debris.

In an implementation the cover has at least one opening for performing measurements. It is beneficial to have one or more openings in the cover so that measurement sensors can measure in one or more directions.

In an implementation the cover is a protective plate above the measurement sensor. It is beneficial to us a protective plate above because it prevents falling rock to hit measurement sensors.

In an implementation the cover comprises openings or is perforated. It is beneficial to have small openings that though which laser measurements can be made but falling rock does not fit through.

In an implementation the measurement sensor is one of the following: a camera, stereo camera, radar, lidar device or thermal camera. It is beneficial to use different types of measurement sensors in the supporting structure.

In an implementation the support structure further comprises at least one support device, wherein the support device is one of the following: a light, flashlight or infrared light. It is beneficial to use the same supporting structure to host additional supporting devices.

In an implementation the measurement sensor is arranged in a forward leaning angle. It is beneficial to use a forward leaning angle for measurement sensors so that they can better see in the front of the mining machine.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings, which are included to provide a further understanding of the visual driving assistance system for a mining machine and constitute a part of this specification, illustrate examples and together with the description help to explain the principles of the visual driving assistance system for a mining machine. In the drawings:

Fig. 1 is a block diagram of an example of a mining machine with a visual driving assistance system;

Fig. 2 is an example of a support structure; and

Fig. 3 is an example of a mining machine with support structures similar to the support structures of Fig. 2.

DETAILED DESCRIPTION

Reference will now be made in detail to the examples which are illustrated in the accompanying drawings.

Fig. 1 discloses an example of a mining machine with a visual driving assistance system. In the example of Fig. 1 the mining machine comprises a first body portion 100 and a second body portion 105. In the example of Fig. 1 the mining machine is proceeding in a mining tunnel in direction of the first body portion 100. In the example of Fig. 1 the mining machine is remote controlled by a person 108. The mining machine may have a cabin that can be used for controlling the mining machine. However, cabin is not necessary if the mining machine is designed for autonomous use or remote use only. In the example of Fig. 1 the controlling is done by the person 108 who has a remote controller, which is configured to communicate with a communication interface 150 of the mining machine. Thus, a cabin is not required in the mining machine of Fig. 1. The remote controller may be any suitable computing device, for example, a tablet computer or a mobile phone. However, it is possible to manufacture a special purpose remote control that is durable and includes display and controls. The communication interface 150 may be a commonly used communication system such as a wireless local area network, blue tooth, local private LTE or 5G network or similar. Instead of a commonly used networking solution a special purpose radio controlling network or similar may be used. The mining machine of Fig. 1 comprises two body portions that are articulated to each other by an articulation joint 110. The number of body portions is not limited to two and it may be three or more. The communication interface 150 may be used for transmitting instructions to the second body portion. This is particularly useful when the second body portion comprises a power train of own and is used driving the mining machine backwards.

The first body portion 100 comprises a bucket 160 and a boom 162 in the front of the mining machine. The front part of the first body further comprises two support structures 164 and 166. In the example of Fig. 1 both of these support structures comprises stereo camera

The stereo camera, which is explained in more detail in Fig. 2, comprises at least two imaging units so that a stereo image can be acquired. Instead of using a stereo camera a two-dimensional camera in both of the support structures could be used for acquiring two images needed for a stereo image. However, in the example of Fig. 1 both of the support structures comprise a stereo camera. In addition to the stereo cameras the support structure may host additional measurement sensors, such as a radar, lidar or a similar. The imaging units of the stereo camera may be calibrated with regard each other and also internally, for example, in order to decrease the lens distortion and other distortions. In the example of Fig. 1 only one stereo cameras of the support structures are in forward orientation, however, there may be more than one so that stereo camera pairs improving the quality can be formed. The stereo cameras in the support structures and other stereo cameras are provided in calibrated locations and preferably attached to the mining machine during manufacturing of the mining machine. Thus, the stereo camera system may be completely calibrated during manufacturing. A recalibration may be performed, for example, when the mining machine is taken into a regular inspection. Furthermore, if the support structures are movable, for example, in upwards, downwards sideways or otherwise, a recalibration may be necessary more often. Movable support structures are equipped with position sensors that can precisely express how the position of the support structure has been changed. Furthermore, the support structure may be only partially movable so that it is attached to permanent positions on the mining machine but, for example, the head part is moving. In such case the position information expresses to movement with regard the moving part or joint. Using this information a processing circuitry or a computer can be used to compute the exact locations of the stereo cameras and other measurement sensors. The processing circuitry 140 may store the latest positions to a nonvolatile memory so that the positions can be retrieved also in case of power loss of the mining machine. If a recalibration is required after a power loss the previous known location may still be useful in order to make the recalibration process faster by using the earlier location information.

In the above position sensors are mentioned, however, the similar position information may be achieved also from servomotors, stepping motors or other actuators that are precise or include an internal position sensor or other means to achieve the same information. The actuator moving the support structure may be so precise that there is no need for continuous calibration process, however, it is possible to use a recalibration method after each movement. When the recalibration is required moving may be preferred only when it is needed.

The stereo cameras acquire stereo images when the mining machine proceeds in the mining tunnel. Mining tunnels are difficult environments as they may be tight and there may be rocks and other obstacles. The stereo cameras are configured to acquire images so that a processing circuitry 140 may generate a three- dimensional model based on the acquired images. The three-dimensional model may be done using the stereo cameras a number of recently acquired images. In addition to generating a three-dimensional model the stereo cameras, when placed on a supporting structure can see behind the bucket 160. Images acquired may be used in order to analyze the content of the bucket and also for increasing working safety. For example, behind a bucket there may be a person working and if an autonomous machine does not detect the person the outcome may be an accident. The same applies also when a person is using a manned mining machine with a cabin. Sometimes it may be impossible to see behind the bucket 160.

In the example of Fig. 1, however, the images from the stereo cameras on the supporting structures are supplemented by additional images from stereo cameras 122, 124, 126 and 130. Stereo cameras 122 and 124 are located on the sides of the first body portion. The locations are calibrated with regard each other and the stereo cameras are preferably attached to these locations already when the mining machine is manufactured. Stereo cameras 126 and 128 are located on the sides of the second body portion. The stereo camera 130 is located on the back and is of particular use when the mining machine is moving backwards. When more than one stereo camera is used the three-dimensional model can be generated immediately and there is no need to wait that the mining machine proceeds along the path so that stereo images acquired by the forward oriented camera would be sufficient in order to facilitate the generation of the three-dimensional model. Furthermore, a plurality of cameras provides a possibility to acquire images from different angles and this improves the quality of the coverage.

In the example of Fig. 1 the processing circuitry 140 is configured to generate a three- dimensional model that is based on the acquired images and the dimensions of the mining machine. The processing circuitry 140 receives position information from the supporting structures 164 and 166 and uses the received information when generating the three-dimensional model. The three-dimensional model comprises the geometry of the mining tunnel and objects that are located in the tunnel including the mining machine itself. The model may include information about content of the bucket and particularly the broken rock material that is collected from the mine. The broken rock is transported to further processing, wherein valuable raw materials are separated from the broken rock. The mining machine is modeled such that the first and the second body portions are in correct locations with regard the tunnel. The modelling is possible because the locations of the stereo cameras are known. Furthermore, the generation of the three-dimensional model may be supported by maps, possibly of high precision, that may be generated using three-dimensional models from one or more mining machines. This is possible because in many parts the mining tunnel does not change over the time but the walls and the roof are stable. When high precision maps or three-dimensional models of the mine already exist it can be used as a support when generating a model of the changing parts, which include other traf fic and other pos sible obstacles . Using the three-dimensional model it is possible to render a three-dimensional view illustrating the mining tunnel. Alternatively, or in addition, the model may be used for facilitating autonomous driving in the tunnel. In such case the rendered view can be used in monitoring the performance of the mining machine.

As explained above the locations of stereo cameras are known because of the made factory calibration. This provides a possibility to combine stereo images acquired by different cameras. Thus, two or more images acquired using different stereo cameras seeing a same object or a feature can be combined in order to improve the accuracy of the three-dimensional model. Even if the example of Fig. 1 shows only one stereo camera per orientation, it should be understood that there may be more stereo cameras and the stereo images acquired by them can be easily combined.

In the example of Fig. 1 the articulation angle can be detected by using a measuring device for measuring the articulation angle of the articulation joint 110. The articulation angle can be used in computing the actual locations of stereo cameras. When the mining machine proceeds along a path in the mine and makes turns, the positions of different body parts change in relation to each other. This naturally causes a similar change in locations of stereo cameras as they move according to the respective body part. When the articulation angle is measured the information can be used for computing the locations of stereo cameras so that one stereo camera in a first body portion can be used together with another stereo camera in a second body portion when acquiring material for three- dimensional model generation.

In the example of Fig. 1 the mining machine comprises a measuring device or indicator for measuring the articulation angle. Instead of using a separate measurement device it is possible to compute the articulation angle using acquired images. Furthermore, there may be separate cameras or sensors that are configured to acquire images of a different body portion. For example, the first body portion may have specific cameras or distance sensors that are configured to measure the distance between different body portions. Then, a distance or a plurality of distance measurements can be used in retrieving the corresponding angle from a database or other data structure capable of storing a look-up table. Thus, any suitable means for estimating or accurately measuring the articulation angle may be used.

In a further example the three-dimensional model generated based on the stereo images is supplemented by using additional image material from one or more additional imaging devices. These additional imaging devices may be two-dimensional digital cameras or thermal cameras. Instead of a two-dimensional camera the stereo camera may be used to acquire two-dimensional images. The mining machine, however, may have additional cameras that are used primarily for other reasons and the additional images may be used to improve the model particularly when objects, persons or parts of the mining machine are blocking the view. When the three- dimensional model of the environment has been created two-dimensional material or thermal measurement can be used to supplement the model. This provides a possibility to show a realistic view of the environment to the driver. In addition to a realistic view an augmented reality view may be shown, wherein the augmented reality view may comprise, for example, driving or navigation instructions. A two-dimensional camera may be associated also with each of the stereo cameras. The two-dimensional camera needs not necessarily to be calibrated similarly as the stereo cameras. The images acquired by the two-dimensional cameras can be adapted to fit the three-dimensional model. This can be done, for example, by detecting the relevant edges or other features and then by mapping the two-dimensional image on the three-dimensional model in order to produce a photorealistic view. This step is naturally not necessary when the two-dimensional images have been acquired using the stereo camera. The photorealistic view is beneficial as the driver perceives the environment as it is.

Fig. 2 discloses two examples of a support structure. The first example comprising parts 200a - 260a is to be used on the right side of the mining machine. The second example comprising parts 200b - 260b is to be used on the left side of the mining machine. The configuration is such that the boom of a mining machine can move freely between the support structures. In the example of Fig. 2 the support structures are identical in functionality with the exception of the intended location, i.e. being opposite hand views. This is only one example and one of the support structures may include components that are not arranged into the other.

The support structures of Fig. 2 comprise a first leg portion 200a, 200b and a second leg portion 205a. The second leg portion 205a, 205b further comprises a protective cover 210a, 210b which may be used to protect, for example, lights that are used to signaling or other electric components. The protective cover 210a, 210b has openings so that lights are visible through it. The upper part of the support structure comprises a stereo camera, wherein a first imaging unit 220a, 220b and a second imaging unit 260a, 260b are spaced apart as required in stereo imaging. In the example of Fig. 2 this space is used for providing a lighting device 240a, 240b in between the imaging units. On top of the supporting structure there is a Lidar unit 250a, 250b for light detection and ranging. The Lidar unit 250a, 250b is used to collect information for three-dimensional model. Lidar unit 250a, 250b is located on the top of the supporting structure and thus may be damaged by falling rocks or if the supporting structure hits the roof for some reason. In order to protect the Lidar 250a, 250b and the supporting structure as such a protective cover 230a, 230b is provided above the supporting structure. The cover is open so that the lidar unit 250a, 250b has full visibility to forward and backward. Furthermore, the cover 230a, 230b has openings like shown in the figure, perforation or other openings that allow laser light at least partially to pass through the cover so that information from the walls and roof of the mining tunnel can be collected. The openings are small enough to prevent larger debris hitting the measurement sensors and the supporting structure as such. The protective cover may be made of metal or hard plastic and it may be easily changed. If there is no need for seeing through the cover, for example, there is no need to monitor the roof the cover may be, for example, a hard metal plate. Instead of a Lidar unit 250a, 250b other components, such as thermal camera, scanner, radar or high-beam light may be used.

Fig. 3 discloses an example of a mining machine having two support structures 320, 330 as discussed above. The mining machine comprises a boom 310 and a bucket 300 and the support structures 320, 330 are located such that they don't limit the movement of the boom 310. The support structures of Fig. 3. are static and do not move, however, as explained above the support structures may be movable in different ways and the number of support structures is not limited to two but there may be one or more support structures. For example, in an implementation there could be support structures of different height so that these support structures do not need to be moved and the mining machine may still acquire different views when necessary.

The above-mentioned arrangements may include computer software which is executed in a computing device, which may be integrated at the visual driving assistance system for a mining machine. When the software is executed in a computing device it is configured to perform the above described inventive method. The software is embodied on a computer readable medium so that it can be provided to the computing device, such as the processing circuitry 140 of Figure 1.

As stated above, the components of the examples can include computer readable medium or memories for holding instructions programmed according to the teachings of the present inventions and for holding data structures, tables, records, and/or other data described herein. Computer readable medium can include any suitable medium that participates in providing instructions to a processor for execution. Common forms of computer-readable media can include, for example, a floppy disk, a flexible disk, hard disk, magnetic tape, any other suitable magnetic medium, a CD-ROM, CD±R, CD±RW, DVD, DVD-RAM, DVD1RW, DVD±R, HD DVD, HD DVD-R, HD DVD-RW, HD DVD-RAM, Blu-ray Disc, any other suitable optical medium, a RAM, a PROM, an EPROM, a FLASH-EPROM, any other suitable + chip or cartridge, a carrier wave or any other suitable medium from which a computer can read.

It is obvious to a person skilled in the art that with the advancement of technology, the basic idea of the driving assistance system may be implemented in various ways. The driving assistance system and its embodiments are thus not limited to the examples described above; instead they may vary within the scope of the claims .