TECHNICAL FIELD

The present invention relates to a method for the inspection and/or maintenance of an electrical panel by means of augmented reality, and to a corresponding system for guiding the inspection and/or maintenance of an electrical panel by means of augmented reality.

In particular, the present invention is advantageously but non-exclusively applied in the inspection and/or maintenance of electrical panels of industrial plants, to which the following description will explicitly refer without losing in generality.

BACKGROUND ART

The electrical panels of industrial plants typically comprise a large number of electrical components connected to one another in a complex manner. In addition, an industrial plant may comprise an equally large number of electrical panels all different in size and combination of electrical components. A maintenance technician who must work on an electrical panel for ordinary or extraordinary maintenance, for example to repair or replace one or more faulty components, needs to know not only what are the possible components for repair or replacement, but also the features of the other components of the electrical panel and the interconnection thereof with the components to be repaired or replaced.

Normally, the maintenance technician who carries out the maintenance of the electrical panel does not have at hand all the information of the electrical panel needed to carry out the maintenance work in an effective and efficient manner, and therefore he/she may make mistakes when interconnecting the replaced components or waste time to find the missing information, searching on paper documents or communicating frequently with an operations center during maintenance.

Systems are known for guiding a technician during the installation or maintenance of industrial products through the use of augmented reality. Such systems include special vision devices specifically constructed which superimpose a virtual information layer to the images of a part of a plant or of a machine to be mounted or serviced. Such vision devices typically include a binocular viewer provided with a video camera, wearable by the technician and permanently connected with a computer configured to generate the virtual information layer. In practice, the known systems for the maintenance of plants based on augmented reality are expensive and particularly difficult to be used. DISCLOSURE OF INVENTION

It is the object of the present invention to provide a method for the inspection and/or maintenance of an electrical panel, through the use of augmented reality, and a corresponding system for guiding inspection and/or maintenance of the electrical panel, which method and system are free from the above drawbacks while being easy to be used.

According to the present invention, a method for the inspection and/or maintenance of a part of an industrial plant using augmented reality and a system for guiding inspection and/or maintenance of a part of an industrial plant using augmented reality are provided as defined in the appended claims.

BRIEF DESCRIPTION OF THE DRAWINGS

The present invention will now be described with reference to the accompanying drawings, which show a non- limiting example thereof, in which:

- figure 1 shows a front view of an electrical panel to be subjected to maintenance;

- figure 2 schematically shows the system for guiding, by means of augmented reality, the inspection and/or maintenance of the electrical panel in figure 1, which system is implemented and operates according to the dictates of the present invention; - figure 3 shows an example of an image of an electrical panel, wherein an intermediate step of the method of the present invention is shown; and

- figure 4 shows an example of a video frame of the electrical panel in figure 1 with superimposed augmented reality information generated according to the method of the present invention.

BEST MODE FOR CARRYING OUT THE INVENTION

In figure 1, reference numeral 1 generally refers as a whole to an electrical panel of the known type, which includes a plurality of DIN rails (not shown in figure 1), and a plurality of electrical components 2, which consist of respective DIN modules installed along the DIN rails so as to form rows 3 of components 2 parallel to a normally horizontal direction 4.

In particular, the electrical panel 1 comprises a frame 5 in the shape of a rectangular frame, which is normally mounted on a wall (not shown) with the major vertical side, and a plurality of rectangular front panels 6, which cover respective DIN rails and are mounted in a known manner on frame 5 so as to be mutually contiguous at the major sides, with the latter parallel to direction 4. In each panel, openings are made which are aligned parallel to direction 4 from which the front parts of respective components 2 mounted on the underlying DIN rail protrude. The electrical panel 1 further comprises a plurality of artificial fiducial elements 8, also known as "markers", arranged in respective distinctive points of the electrical panel 1. For example, figure 1 shows four fiducial elements 8 applied outside the four corners of frame 5, so as to be clearly visible from the front of the electrical panel 1, and each consisting of a respective adhesive element having circular shape and a color that contrasts with the color of frame 5.

According to a variant of the invention, each artificial fiducial element is printed on the outer surface of frame 5.

According to a variant of the invention, each fiducial element consists of a specific DIN module, which is mounted on a DIN rail in a specific position as any of the other components and is passive, i.e. has a graphic sign which is easily identifiable, or of the active type, i.e. has a bright element, for example an LED.

In figure 2, reference numeral 10 generally indicates the system for guiding the inspection and/or maintenance of the electrical panel 1 by means of augmented reality, implemented and operating according to the present invention .

With reference to figure 2, system 10 includes a memory 12 to store a database 13 containing information _r

6 relating to all the components installable in the electrical panel 1 and which includes in particular, for each of the installable components, respective geometric data, respective visual information to be displayed with augmented reality, and a respective numerical descriptor indicative of certain local visual features of the component. The geometric data contain the dimensions and the geometric proportions of the corresponding component. The visual information includes: technical specifications, and/or function, and/or logical group, and/or interconnection modes of the corresponding component. The numerical descriptor of each component is a vector of numerical values calculated on an image of the component according to known techniques.

According to a preferred embodiment of the present invention, the numerical values of each numerical descriptor relate to visual features of the component sampled in a plurality of points of the component image. The sampling points are the vertices of a regular mesh grid or are located in significant regions of the component, such as the corners of the component body. In the neighborhood of each sampling point, numerical values are calculated which are representative of some features of the image, such as image gradient (module and phase) , response to 2D filters, color components, etc. Therefore, a numerical descriptor is representable in principle as an array of numerical values related to visual features of the component and calculated in a neighborhood of each one of multiple points of an component image. Assuming that the number of sampling points is N and that the number of visual features calculated for each point is M, the descriptor is represented as an array of N x M numerical values, which for convenience of calculation is stored as a vector of N x M numerical values.

System 10 also comprises an installation device 14, which consists for example of a personal computer or a laptop computer capable of acquiring data from database 13, and a mobile inspection device 15, which typically consists of a smartphone or a last-generation tablet computer and is therefore provided with a respective digital video camera and a respective display screen, preferably consisting of a touchscreen. Memory 12 consists, for example, of a network disk forming part of a remote data network or is arranged inside a remote server. In both cases, the installation device 14 must be provided with its own wired or wireless communication means in order to connect to the remote data network or remote server. According to a variant (not shown) of the present invention, memory 12 is an internal component of the installation device 14.

System 10 also comprises a description device 16 (figures 1 and 2) , which is adapted to be mounted on the electrical panel 1 and comprises a memory 17 for storing information related to the electrical panel 1, as will be better explained below, a control unit 18 and a communication module 19 to communicate with the mobile inspection device 15 and, optionally, with the installation device 14. The description device 16 consists for example of a single RFID integrated chip to communicate via radio through known standard radio protocols, for example the NFC (Near Field Communication) standard, with similar RFID modules integrated in the devices 14 and 15. Alternatively, the description device 16 is specifically constructed, and in particular the control unit 18 consists of a microcontroller of industrial type and the communication module 19 includes an RFID interface and/or a Wi-Fi interface and/or a serial interface (RS-232) .

System 10 also comprises, as part of one of the devices 14 or 15, or as a stand-alone device, an image acquisition device consisting, for example, of a digital camera of a known type, and thus not shown, for acquiring at least one image of the electrical panel 1.

The installation device 14, the mobile inspection device 15 and the description device 16 comprise respective software objects, in the case of devices 14 and 15, or firmware, in the case of device 16, stored in the respective memories and designed to implement, when executed on the respective processing or control units, the following steps of the inspection and/or maintenance method of the present invention.

When assembling or initially testing the electrical panel 1, or when a service technician is about to make an inspection and/or maintenance of the electrical panel 1 for the first time, the description device 16 is subject to an installation step, in which the technician acquires, using the image acquisition device, at least one digital image IM of the front view of the electrical panel 1 and transfers image IM to the installation device 14, which is programmed to create a general model MD of the electrical panel 1 on the basis of the image IM received and of the information contained and acquired from database 13. The general model MD includes a geometric model MG of the electrical panel 1 and the visual information related to components 2 installed in the electrical panel 1.

In particular, the software with which the installation device 14 is programmed allows the latter to recognize the components 2 in image IM on the basis of all the numerical descriptors acquired from database 13 and determine, on the basis of the geometric data of the components 2 that have been recognized, the position coordinates, referred to image IM, of the recognized components 2. The geometric model MG created consists of the coordinates, referred to image IM, of the position of the recognized components 2.

The recognition of the components 2 in image IM is performed by calculating, for different areas of image IM, respective numerical descriptors of the same type as the numerical descriptors stored in database 13, and then comparing each numerical descriptor calculated with at least part of the numerical descriptors stored in database 13 to find, among the latter, one which is equivalent to the calculated numerical descriptor. In other words, if the numerical descriptor calculated in a certain area has a high similarity with a numerical descriptor stored in database 13, then it means that it is highly likely that in that area there is the component with which that numerical descriptor stored is associated. The comparison between two numerical descriptors consists, for example, in calculating the Euclidean distance in the space at N x M dimensions, between the two vectors which define the two numerical descriptors.

In the particular case of applying the invention to the electrical panel 1, the recognition of components 2 occurs after a preliminary identification, in image IM, of horizontal linear image portions in which there are the rows 3 of components 2. With reference to figure 3, which shows an example of the image IM of an electrical panel 1 in which there are panels 6 without components 2, the linear image portions to be identified are indicated with reference numeral 9 and substantially coincide with narrow strips of image IM parallel to the direction 4, in a direction 7 orthogonal to direction 4, around components 2.

More in detail, a scan of image IM is performed to record the image contrast variability rate in direction 4 upon the variation of the position along direction 7, according to calculation techniques known in the field of artificial vision. The intervals of positions along direction 7 in which the image contrast variability rate exceeds a certain threshold value are then searched. Indeed, in the positions along direction 7 corresponding to image areas where there are no components 2, the contrast variability rate is necessarily very low. The image portions 9 are thus defined by the positions along direction 7 in which the image contrast variability rate exceeds the threshold value. Once the image portions 9 have been identified, a numerical descriptor is calculated in each of a plurality of areas within each image portion 9 by using a constant sampling step in the direction 4 having a value from 2 to 10 pixels.

It is worth noting that searching components 2 throughout image IM in a comprehensive manner is certainly possible but is expensive from the computational point of view, since it would be necessary to calculate numerical descriptors in a large number of areas of image IM. The identification of the image portions 9 allows discarding most of the non-significant areas of image IM, thereby focusing the calculation of numerical descriptors only in the areas where it is more likely to find components 2, thus reducing the processing time of the installation device 14.

Model MD is transferred from the installation device 14 to the description device 16 to be stored in memory 17. The transfer of model MD from the installation device 14 to the description device 16 takes place indirectly by means of a first data transmission from the installation device 14 to the inspection device 15 and a second data transmission from the inspection device 15 to the description device 16, or directly by means of a temporary connection established during the single installation step, for example via a serial cable RS-232, between the installation device 14 and the description device 16. At this point, the step of installing the description device 16 is completed and the model MD of the electrical panel 1 is available for the next steps of inspecting and/or maintaining the electrical panel 1. Considering that the number of descriptors stored in database 13 is potentially large and that each of the installable components can be mounted in different positions inside the electrical panel 1, and despite the fact that the numerical descriptors are calculated only in certain areas within each of the image portions 9, the time required for the step of installing the description device 16 is not negligible. However, it is a one-off step and therefore does not affect the overall convenience of the method.

For each inspection and/or maintenance event, the technician controls the inspection device 15 so that it acquires the model MD of the electrical panel 1 from the description device 16 and loads it onto its memory. The technician places the inspection device 15 so that the latter focuses with its digital video camera on the electrical panel 1 in a substantially front view. The inspection device 15 automatically carries out, after loading model MD or on explicit subsequent control entered by the technician, a continuous acquisition of video frames of the electrical panel 1. The controls are entered via the touchscreen or keyboard provided with the inspection device 15. The inspection device 15 is programmed to display in real time at least part of the visual information contained in the model MD as a function of the geometric model MG and in a manner consistent with the real-time display of the video frames to guide the technician during the inspection and/or maintenance event. In particular, the software with which the inspection device 15 is programmed allows the latter to carry out the following processing operations.

First, the inspection device 15 analyzes each video frame to detect in real time the position of fiducial points on the electrical panel 1 in the video frame. In particular, the inspection device 15 searches for fiducial elements 8 in the video frame and determines the position coordinates thereof referred to the video frame. Knowing the fiducial points in the video frame allows the inspection device 15 to infer its distance and orientation with respect to the electrical panel 1.

The inspection device 15 then determines in real time, in each video frame, the position of the components 2 as a function of the geometric model MG and of the position of the fiducial points in the video frame. In particular, the inspection device 15 calculates the position coordinates of the components 2 by means of a perspective transformation of the position coordinates of the components of the geometric model MG, as a function of the position coordinates of the fiducial points in the video frame. The perspective transformation, also known as "image warping", allows locating, in the video frames, the components of the geometric model MG even if the video shooting angle varies.

Finally, the inspection device 15 superimposes in real time at least part of the visual information extracted from model MD to each video frame as a function of the position of the components 2 in the video frame. The inspection device 15 displays on its screen the sequence of video frames in real time with the corresponding visual information superimposed, thus creating an augmented reality display of the electrical panel 1 which guides the technician during inspection and/or maintenance.

The inspection device 15 optionally manages the superimposition of the visual information to the video frames in an interactive manner, to allow the technician to select, for example via the touchscreen or keyboard of the inspection device 15, the components 2 for which he/she wishes to display the information.

Figure 4 shows an example of augmented-reality display of the electrical panel 1 in figure 1, in which the information relating to two components selected by the technician and indicated with reference numerals 2a and 2b is displayed. The information includes first information concerning the logical group to which components 2a and 2b belong, indicated with DEVICE DEVICE-A and-B, and second information concerning the operating status of the two components 2a and 2b, indicated with STATUS-A and STATUS-B. Figure 3 also shows a label, indicated with DATASHEET, which is clickable to access one or more files outside the model MD with more detailed information on the technical specifications of components 2 in the electrical panel 1.

When inspecting and/or maintaining the electrical panel 1, the technician may repair or replace any faulty component 2 installed or simply note that the installed component 2 does not work properly. Any change to the electrical panel 1, or any indication of a fault of some components 2 must be recorded in the description device 16. For this purpose, the technician inputs the information relating to the replaced or faulty component in the inspection device 15 and the inspection device 15 transmits it to the description device 16 so that the latter stores it in memory 17 among the visual information of such a component .

According to a further embodiment of the present invention, the electrical panel 1 is free from the artificial fiducial elements 8 in figure 1 and the installation device 14 is programmed to identify, in the image IM, the fiducial elements of the electrical panel 1 and include the geometry of the fiducial elements in the general model MD to be stored in the description device 16. The fiducial elements to locate in image IM consist of particular distinctive elements which characterize the front view of the electrical panel 1, for example the outer or inner edges of frame 5. When inspecting and/or maintaining the electrical panel 1, the inspection device 15 detects the fiducial points in each frame as a function of the geometry of the fiducial elements contained in the model MD .

The advantages of the method for the inspection and/or maintenance of an electrical panel by means of augmented reality provided by the present invention appear clearly from the above description. In particular, the augmented- reality inspection and/or maintenance method and the corresponding system 10 for guiding the inspection and/or maintenance by means of augmented reality allow the service technician to have access, in real time during the inspection and/or maintenance operation, to the features of all components 2 of the electrical panel and to have information often difficult to be found, especially if the service technician is not sufficiently expert in the specific electrical panel 1.

Finally, it should be noted that the inspection and/or maintenance method of the invention is adapted to the inspection and/or maintenance of any part of an industrial plant comprising a plurality of components installed therein .