BACKGROUND OF THE INVENTION

A. FIELD OF THE INVENTION

The present invention is related to simulation systems and methods and more particularly to a system and method for displaying a virtual industrial machine in an immersive virtual space.

B. DESCRIPTION OF THE RELATED ART.

The digital transformation demanded worldwide by the IV Industrial Revolution increasingly requires more efficient and flexible technological platforms for training and education. An engineering education based on modem technologies is the key to these future platforms. To this end, a wide variety of schemes have been developed, such as virtual laboratories, online courses and websites, most of them dedicated to the transmission of knowledge. However, in the field of Engineering, not only the transmission of knowledge is of great importance, but also the development of skills and knowledge that can only be obtained by applying such knowledge in practice. Since the use of a laboratory in Engineering Education is an essential means to achieve a higher level of learning and to be able to develop the skills and abilities required to put knowledge into action, the laboratory experience cannot be left aside. A novel and highly effective alternative to support learning, face- to-face and distance training in engineering are the technological platforms of Remote and Cyber Physical Laboratories.

For example, US patent application No. US20140156234A1 describes a method for creating a simulation of an industrial automation system comprising: creating a simulation of an industrial control system at least in part using a simulation creation module on a computer: modeling industrial modules at least in part using the simulation creation module on a computer; creating communication channels between the modeled industrial control module and a physical industrial control module at least in part using a simulation execution module; and synchronizing the timing between the simulation and the physical industrial control module at least in part using the simulation execution module. US patent No. US10360316B2 describes methods and systems for creating and running and industrial control system simulation. The simulation may include animation of a complex machine linked with the industrial control device controlling the complex machine. The simulation may also include links to the physical I/O and other modules of the industrial controller to enhance the functionality of the simulation. This may increase the likelihood that the timing and functionality of the simulation may be more like real-time operation of the industrial control system. This may enhance the system design and save time of system design and start-up and troubleshooting of the operational industrial control system.

The main disadvantage of prior art systems and methods are the following:

• Prior art systems are complex, costly, impractical, and do not reflect what happens in an industrial environment.

• The prior art systems do not provide real-time simulations because they contemplate the connection between a controller, a simulator and a display, so data exchange is slow.

• The process by which the simulation of the automation system is generated is not practical and does not reflect what happens in an industrial environment, since in practice all these tools do not have to be integrated.

• The simulation of the machines provided by the methods and systems of the prior art is not complete since the simulations only provide the visual appearance of the machines.

In view of the above referred problems, applicant developed a method and system to provide a fully immersive virtual environment through the projection of a virtual reality machine which is called a Cyber-Physical Lab (CPL).

Cyber-Physical Labs (CPL) are a combination of physical laboratories with real controllers and virtual reality machines (VRM) to emulate automation and control processes. CPLs are platforms developed for the training of engineers in automation, control, and robotics, which meet aspects such as low cost and effective use of laboratories. In a Cyber-Physical Lab, physical devices and virtual components are fully connected working together, each operating in their own spaces, real and virtual, but interacting with each other. Cyber-Physical Labs (CFL) provide a solution to fully equip engineering laboratories, reducing costs without sacrificing functionality or efficiency. A CFL is a laboratory equipped with real industrial devices to control processes and Virtual Reality Machines to emulate real manufacturing processes or systems. A virtual reality machine (VRM) is a computational application that emulates, by software, the appearance and behavior of real machines. It is based on a set of 3D solid models that mimics, as closely as possible, the visual, audible and functional aspects of the actual machine elements. However, the sensors and actuators of the virtual model are connected to the addresses of a real controller. Thus, the dynamics and functionality of the Virtual Reality Machines depend solely on the logical control sequences programmed in the external physical controller.

On the other hand, a fully immersive virtual environment for the cyber physical lab which comprises full size (floor to ceiling) projection screens that surround the users, significantly enhances the learning process. Its implementation increases the functionality of the lab significantly, building a fully immersive Cyber Physical environment with physical devices and virtual components working together, each operating in their own spaces, real and virtual, but interacting with each, other. From a pedagogical point of view, the fully immersive virtual environment significantly increases the overall understanding of students, since a fully immersive environment created through technology to emulate reality, uses scenarios, objects or figures, equipment and instruments that appear real, so the user enjoys an immersive experience.

Other advantages of the method and system of the present invention are:

• The generated simulation is in real time because the virtual reality machine connects directly to a controller using a native communication protocol of the controller, so the data exchange is in real time.

• The method and system of the present invention is closer to what happens in a real automation environment, since only the programming environment, the controller and the virtual reality machine are required. • The virtual reality machines emulate the full static and dynamic behavior, and the visually and auditory appearance, of the real machines.

• The Virtual Machine connects directly with the controller using a native communication protocol of the controller so that the data exchange is in the real time of the machine.

• The Virtual Machine connects directly with the controller and using a native communication protocol of the controller so that the data exchange is in the real time of the machine.

• Cyber-Machines emulate the full appearance of machines, both visually and auditory, and the static and dynamic behavior of the real machines.

SUMMARY OF THE INVENTION

It is therefore a main object of the present invention to provide a method and system for generating a fully immersive virtual environment through the projection of a virtual reality machine which is called a Cyber-Physical Lab (CPL).

It is also a main object of the present invention to provide a method and system of the above referred nature in which the virtual reality machine (VRM) emulates, by software, the appearance, both visually and auditory, and the static and dynamic behavior of the real machines.

It is an additional main object of the present invention to provide a method and system of the above referred nature which generates a simulation in real time thanks to the virtual reality machine connecting directly to a controller using a native communication protocol of the controller, so the data exchange is in real time.

It is also an additional object of the present invention, to provide a method and system of the above referred nature is closer to what happens in a real automation environment, since only the programming environment, the controller and the virtual reality machine are required.

It is another main object of the present invention to provide a method and system of the above referred nature in which the fully immersive virtual environment full size (floor to ceiling) projection screens that surround the users, significantly increases the overall understanding of students, since a fully immersive environment created through technology to emulate reality, uses scenarios, objects or figures, equipment and instruments that appear real, so the user enjoys an immersive experience.

These and other objects and advantages of the method and system for generating a fully immersive virtual environment of the present invention will become apparent to those persons having an ordinary skill in the art, from the following detailed description of the embodiments of the invention which will be made with reference to the accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

Figure 1 is a diagram of the system for generating a fully immersive virtual reality environment showing its physical components and the physical connections between them.

DETAILED DESCRIPTION OF THE INVENTION

The typical system of the present invention is comprised by physical components and logical components, wherein the system of the present invention comprises:

Physical components (hardware) a VRM computer (1) having a high performance graphic card (2), for running one or more virtual machines; projection means (3), connected to the graphic card (2) of the VRM computer (1), said projection means (3) comprising one or more high performance projectors for projecting an image of the one or more virtual machines on surfaces having a preferred size of 5X5X3 (length-width-height). High performance means that the projector has crystal lens that do not produce any distortion in the projected image and allows to project a image from short distances typical of a laboratory. One example of such projectors is the Epson Powerlite PRO L1070U having an Epson lent ELPLX01 (V12H004X01). working stations comprising: automatization equipment, comprising: one or more programmable controllers (4), each controller (4) connected to the VRM computer (1) processing the virtual reality machines for controlling the operating logic of virtual reality machines, specifically, each controller (4) receives and processes both digital and analog input signals, executes user's programming code, and generates output signals that are sent to the VRM computer (1 ) running the one or more virtual machines. Preferred specifications for a controller (4) comprises: CPU 1516-3-PN/DP; work memory 1 MB code and 5 MB data; 10 ns bit instruction time; 4-stage protection concept, integrated technology functions: Motion Control, closed-loop control, counting & measuring; integrated tracing; 1st interface: PROFINET IO controller, supports RT/IRT, 2 ports, MRP, transport protocol TCP/IP, S7 communication, Web server; 2nd interface: PROFINET basic services, transport protocol TCP/IP, Web server; 3rd interface: PROFIBUS DP master, constant bus cycle time; firmware V1.8; one or more HMI (Human Machine Interface) devices (5), each one connected to a respective programmable controller (4), and each HMI device (5) comprising a device having a graphic interface with which a user can monitor and operate the status of an automation process carried out by a correspondent virtual reality machine and additionally takes or performs control actions or process operations, system development computer (6), connected to each programmable controller (4), for running controller programming environment (STEP 7 Professional) that allows programming of controllers for the automation of the machines (virtual reality machines). It also runs the following development systems which will be described later: Simatic TIA Portal and WinCC Advanced; physical space: comprising a room having a minimum preferred dimension of 5mX5m and a minimum height of 3m, wherein the walls comprise the projection surfaces. In other embodiments of the invention, the programmable controllers are implemented as virtual controllers (VC) that may be processed by the VRM computer (1). In such embodiment, the virtual controllers (VC) are not physical components but logical components.

Logical components (software) virtual reality machines: one or more virtual reality machine (VRM) modules: virtual machines are computational emulations that reproduce the behavior of a real machine or process with all its static and dynamic characteristics, as well as their respective space and time constraints. Each virtual machine is run by the VRM computer and receives the output signals produced by the programmable controller. In a preferred embodiment, the system comprises the following virtual reality machines: sorting line VRM: emulates a storage tower that has a conveyor chain that, in turn, is connected to a conveyor belt, both electrically operated by a gear system. In addition, the sorting line VRM emulated three electromechanical pistons and four unloading tables, each with a light reflection sensor to identify that the part has arrived at its destination or the presence of a part on the work table; processing line VRM: emulates a three-head machining tool, three bi-directional conveyor belts, a turntable with bi-directional conveyor chain and a supply table. Each of the conveyor belts and the turntable have an emulated inductive proximity sensor to detect the presence of workpieces; cartesian robot VRM: emulates a Cartesian coordinate robotic arm that can perform movements in three linear directions (X, Y, Z) and an electromagnet, a set of parts and an unloading station or miniwarehouse. The movement limits of the robotic arm in the three axes are detected by emulated mechanical contact sensors; elevator model VRM: emulates a four-story tower, a cabin that is electrically driven by a system of pulleys and gears, sliding doors that are activated pneumatically, buttons and indicators on each floor, and lamps to indicate the position and direction of movement of the elevator. It also has a control panel, which emulates the options available inside an elevator; virtual reality machine connection and interface program : this component is run by the VRM computer (1) and allows to: select and execute a VRM, select the kind of communication protocol to be used between the VRM and the programmable controller and the logical direction of the VRM, visualizing the operation of the VRM on the display of the VRM computer, configuring the operating conditions of the VRM such as adding or retiring manufacturing parts, restarting the operation of the VRM, detecting the position of sensors, detecting parameters of the sensors and all the relevant information about the operation of the VRM ; development systems:

Simatic TIA Portal: automation project management tool, Integrates and manages the rest of the applications (STEP7 and WinCC) in a single development environment;

STEP 7 Professional: tool for the configuration and programming of the programmable controllers for the automation of the virtual reality machines and/or processes;

WinCC Advanced: tool for the configuration of HMI devices and the design of Human Machine Interfaces for the operation and monitoring of automation processes;

Connection between the virtual reality machines and the programmable controllers

Physical connection

As previously described, the virtual reality machines (VRM) are 2D or 3D models having a high definition and high level of detail. The virtual model is not software, it is an executable application that does not require installation for its use. Since the VRM’s have the same (virtual) actuators and (virtual) sensors considered in a real industrial process they can be connected via PROFINET, PROFIBUS or any other industrial network to the programmable controller.

Therefore, the VRM computer (1) must have installed a communication card (7) selected from the group comprising but not limited to: MPI, PROFIBUS, PROFINET, Ethernet (TCP/IP) or Siemens PLC emulator, PLCSIM to be able to connect to the programmable controllers.

If the controller comprises a virtual controller, then instead of a communication card a virtual communication module (8) run by the VRM computer (1) or any other computer may be used.

Logical connection

Depending on the communication card (7) installed in the VRM computer (1) (or the communication module (8), when the VRM is loaded and executed by the VRM computer (1 ), it is given a specific logic address. Then, by means of the virtual reality machine connection and interface program, it is selected the kind of communication protocol with the programmable controller (4), (VC), and the logic address of the VRM to make the logic connection between the programmable controller (4), (VC) and the VRM.

The virtual reality machine connection and interface program also allows to monitor the connection status between the programmable controller (4), (VC) and the VRM, i.e. successful connection or an error in the connection.

It important to verify that the kind of communication protocol used between the Simatic TIA Portal, and the programmable controller (4), (VC) matches the one selected for the VRM in case the virtual reality machine connection and interface program shows and error in the connection.

The connection between the VRM and a simulated (VC) or real programmable controller (4) is achieved by a set of software communication libraries. Whether the programmable controller comprises a physical component (4) or a virtual component (VC) the connection process is the same since the connection is carried out between the IP direction assigned to the physical or virtual controller. Depending on the desired connection (whether with a simulated or with a real programmable controller), an appropriate library is used. Besides, when working with real programmable controllers (4) , the libraries allow the connection through almost all the programmable controller (4) supported industrial networks, such as MPI, PROFIBUS, and Industrial Ethernet. In the preferred embodiment of the invention, the libraries are divided in five groups according to the LabVIEW’s programming standard: configuration, open, write, read, and close section. The write and read groups establish the writing or reading of the controller's memories, where the state of the sensors and actuators of the virtual reality machine to be used will be stored. The open and close groups open or close the communication. The configuration group set the parameters to connect to the controller, such as what type of network will be used and what type of communication (PROFIBUS, ETHERNET, MPI).

The libraries also take advantage of the polymorphic property, in order to write or read different data types with the same write or read function.

The VRM’s receive the control signals from the correspondent controller (4), (8) trough the communication card (7) installed in the VRM computer or through the communication module (8). The control signals are received by the VRM's without any kind of data conversion, compression, encryption or any kind of modification. Programming of the Virtual reality machine

For the development of a VRM, it is used LabVIEW, a graphic programming software specialized in the development of data acquisition and instrumentation applications. Previously, the different components of the machine are modeled in 3D with the use of a CAD-type package (for example SolidWork). 3D models are exported to a VRML format, which allows LabVIEW to read, open, and display the components in a 3D scene. The assembly of the parts is done in LabVIEW, as well as the logic programming for the movement of the parts, depending on each I/O (input/output) of the PLC through the communication protocol that is enabled both in the PLC and on the VRM computer. The result is a model of a 3D Virtual Reality Machine , which is identical to the real machine in appearance and functionality.

Programming and configuring the system

The programming of the controllers (4), (VC), the connection between the programmable controllers (4), (VC) and HMI, the creation of projects and the configuration of the HMI's is carried out in the same way it is done for applications using real machines and is known in the art.

Operation of the Virtual reality Machine by the user a) powering up a programmable controller (4) or start the processing of a virtual controller (VC) and a HMI (Human Machine Interface) (5) device corresponding to the VRM machine to be operated; b) begin the execution of the virtual reality machine connection and interface program by means of the VRM computer (1); c) begin the execution of a VRM selected from a list shown by the virtual reality machine connection and interface program. Once the VRM is running, the connection between the graphic card (2) of the VRM computer (1) and the projecting means (3) is carried out by the virtual reality machine connection and interface program to begin the projection of the VRM by the projecting means (3). The VRM itself, controls the projection of the Virtual Reality Machine wich is processed by the VRM computer (1). The display process by the projecting means (3) is the same process that any computer does to display information using a graphic card (2). d) perform the logical connection between the VRM and the programmable controller (4), (VC) using the virtual reality machine connection and interface program, comprising the steps of: selecting the kind of communication protocol with the programmable controller (4), (VC) and the logic address of the VRM to make the logic connection between the programmable controller (4), (VC) and the VRM ; e) verify that the connection between the VRM and the programmable controller (4), (VC) is successful by means of the virtual reality machine connection and interface program; f) configuring the operating conditions of the VRM by means of the virtual reality machine connection and interface program. g) operate the VRM projected by the projecting means using the HMI (Human Machine Interface) device (5) in the same way that a real machine is operated. It is important to note that the methods and hardware used for performing the connection and communication between the programmable controllers (4), (VC), the HMI devices (5) and the system development computer (6) are known in the art and are the same that are used for controlling real machines.

Finally, it should be understood that the simulation system and method of the present invention is not limited to the embodiments described above and that those skilled in the art will be enabled, by the teachings herein established, to make changes in the simulation system and method, of the present invention, which will fall within the true scope of the invention, the scope of which will be exclusively established by the following claims.