FIELD OF THE INVENTION:

[0001] This invention relates to robotic control, and more particularly to an enhanced reality underwater maintenance system by using a Virtual Reality Manipulator (VRM) to replace the human diver presence underwater and perform all the required tasks with same quality.

BACKGROUND OF THE INVENTION:

[0002] offshore diving is a known term of divers working in support of the production sector of the offshore oil and gas industry, this includes construction and maintenance of all subsea assets in general.

[0003] offshore air diving services are usually limited to a maximum water depth of 40 meters below sea level and limited time of around 90 minutes per diver per day.

In deeper waters, saturation divers can go up to approximately 300 meters, but very expensive equipment and gases are required (around 10 times the cost) and much higher risks to lose human lives. In deeper waters over 300 meters, very advanced equipment and techniques are used with a Work Class Remote Operated Vehicle (WCROV) presence to replace the process done by divers. This advanced equipment and the limitation of not having the diver flexibility increases the cost of similar activities in shallow waters up to 30 times the usual air diving process.

[0004] It is worth mentioning that using a Work Class Remote Operated Vehicle (WCROV) in higher depths (higher than 30m) has the advantage of going to any depth and work for unlimited time, but the WCROV operator has the following limitations when compared to divers:

• The operator can only see 2D vision on a screen from a HD camera which limits the capability and accuracy of the WCROV manipulators usability where the diver can see 3D vision to estimate the depth and size to operate properly the diver’s hands. • The manipulator number of joints and dimensions is different from the human arms where human gather the skills to move the arms and legs over years of accumulated trials while we were babies learning how to use our arms, these trials are stored in our back mind, such experience is missing and need years to build for a WCROV operator to be able to use efficiently the manipulators as accurate and fast as his arms.

• Cannot feel force or torque on the manipulator as the diver’s hands, therefore cannot estimate if he has a good grip on a tool or the resistance while tightening a nut.

[0005] The Remotely Operated vehicles (WCROV) are robots with go to great depths underwater and typically controlled from a surface vessel using a joystick, tether connects WCROV to vessel. Additional equipment, such as a manipulator, different tools to make different tasks ex: cutting, welding or giving hydraulic power to other tools. Manipulators are fitted on WCROV front or end to perform subsea work. Remote manipulation allows human operators working from the surface to perform different tasks like plugging and unplugging electrical and hydraulic connectors, due to 2D vision generated from navigation cameras equipped on WROV, most tasks are done by special dedicated tools which are heavy, expensive and takes a long time because of the

WCROV pilot limitation as he cannot see 3D to sense the distance and dimensions between different objects, cannot move the manipulator arm easily or accurately and cannot sense any feelings.

[0006] The present invention provides an enhanced reality maintenance underwater system by using a Virtual Reality Manipulator (VRM) which will enable the WCROV to replace the human diver presence underwater and perform all the required tasks with a better quality because of:

• 3D Vision:

WCROV manipulator is equipped by multiple cameras underwater with MCS Video Codec s/w to develop video feeds through the human eyes to the human brain which managed easily to interpret the video feed and simulate a 3D vision environment where the human can feel depth, distances and sizes through a Virtual Reality Helmet. Moving Manipulators as if his own arms:

When the human operator moves his arms holding motion sensors in his hands, the result of the movement is sent Realtime to the MCS Manipulator Movement Artificial Intelligence (MMAI) s/w which acts as the human back brain and solve the movement of each joint of the 7 joints of the manipulator for its fingers position and direction to match the human hand.

The manipulator hand will be exactly where the brain wants it to be when the human hands were moved.

• Sensing through Force and Torque:

System is equipped by force and torque sensor, the readings coming from force and torque sensor enables system to sense the required actions needed to perform each maintenance task.

• 3D Measurements:

Due to presence of PRC module, VRM solution is able to obtain accurate measurements of sub sea assets from generated 3D models.

Accordingly system is able to take better decisions based on these readings.

Disclosure of the invention :

summary of the invention:

[0007] The present invention relates to an enhanced reality maintenance underwater system by using a virtual reality manipulator which is an advanced solution consists of ROV manipulator arms , Virtual reality helmet and controllers , 3D cameras , force and torque sensors all integrated together to replace the human divers underwater providing same quality.

The Manipulator system of the present invention consists of the following:

1. VR

a. Helmet

b. Controllers 2. AI

The AI is used to move the arm with all of its 7-joints in a correct setup to make the manipulator get to the target desired by the user with the correct orientation.

It runs in Realtime to figure out the 7-joints angles to match the desired target and orientation. It uses different AI algorithms such as neural networks and Genetic algorithms.

[0008] the components of Virtual Reality Manipulator (VRM) system consists of the following:

1. Stereo cameras.

2. Virtual Reality solution.

3. Force/Torque sensor on each manipulator

4. Manipulator Movement Artificial Intelligence MMAI

5. Photo Realistic Cloud (PRC) Solution.

[0009] the method of operating the Virtual reality manipulator VRM of the current invention comprising the below steps:

1 -Firstly, the user is required to wear a VR helmet on his head to see a live 3D video of the object(s) that he want to interact with as well as the robotic arm.

2-The user also has to hold both VR controllers, one in each hand, the VR controller will transfer the user wrist movement or rotation.

To the mechanical arm so any movement that the user does with his wrist will be emitted and replicated by the mechanical arm.

3-After the user wear both VR helmet and VR controllers, the user can consider the robotic arm(s) as his own and begin to interact, move and rotate his wrist to virtually interact with the objects and the robotic arm will replicate his movements which will interact with the objects in real world.

/Torque Sensor System. Detailed Description of a Preferred Embodiment;

[0011] the enhanced reality maintenance underwater system of the current invention consists of the following elements:

(A) Manipulator system consists of virtual reality helmet (1) , controllers (2) and Artificial Intelligence, wherein Artificial Intelligence (AI) is used to move the arm with all of its 7-joints in a correct setup to make the manipulator get to the target desired by the user with the correct orientation.lt runs in Realtime to figure out the 7-joints angles to match the desired target and orientation. It uses different AI algorithms such as neural networks and Genetic algorithms.

(B) 3D Cameras are installed on the WCROV Manipulator to produce the three- dimensional environment, cameras are installed on pan and tilt unit (8), when operator moves the helmet (1) in any direction or angle this move is translated through software to move the pan and tilt unit (8) and accordingly to move camera the desired direction or angle. Exactly like human moving his head to see something. Such flexibility enables operator to explore the underwater environment exactly like he is in it.

The stereo cameras (6) have specially made software that could zoom in and out in 3D environment to populate a similar vision to these interpreted by human brain through human eyes.

[0012] (C) Virtual Reality solution, it consists of virtual reality helmet (1) and controllers (2) WCROV Operator wears the helmet on the vessel top to view the three- dimensional environment underwater, this view is generated from the 3D cameras mentioned in the previous point, Operator holds controllers (2) in his hands to move WCROV manipulator arms (4) accordingly, the result of the movement is sent Realtime to Manipulator Movement Artificial Intelligence to move the manipulator arms, the controllers can perform free moves on manipulator’s 7 joints to perform different tasks. [0013] (D) Force/Torque sensor (9) on each manipulator

Developing Force/Torque sensor to work underwater in full ocean depth without affecting its sensitivity to feel Force / Torque in all 6 directions.

Customizing the WCROV manipulators with the F/T sensor and integrating it with the extra extension.

It is worth mentioning that the ROV manipulator arm has been modified to allow the installation of a force / torque sensor inside and an underwater pod created for it and cabling / wiring has been implemented so the force and torque generated from manipulator arms due to its different actions can be measured, these measurements are passed to artificial intelligence module and consequently understood.

Every manipulator arm is equipped with 1 force and torque sensor.

These data appears to the person wearing the virtual reality helmet so he/she can take decisions upon , understands if the force he is using is excessive to break the object , use more power to tighten bolts...etc

Also Measurements from force / torque sensor are stored into the AI module, these measurements are understood by this software module and affects the decision making of the solution in automatic mode.

[0014] (E) Manipulator Movement Artificial Intelligence MMAI

MMAI is a software solution based on artificial intelligence technology, it’s the replacement of the diver’s back brain to move his arms for his hand to be in a certain position.

When the human moves his arm, the MMAI will compute the best setting for each joint of the manipulator to result that the manipulator hand’s position will be in the same position as the human hand in the virtual reality environment the human brain sees, then the MMAI will execute such settings to the manipulator joints simultaneously to maintain the same path of the human hands while reaching the final position (inserting a bolt in a flange hole, the manipulator hand must move in a certain direction for the bolt to be installed). The system takes the torque/force readings to enable the AI module to achieve a task just like a human brain use the fingers feeling to sense accurately where is a hole in a flange to insert a bolt, therefore the MMAI is capable to automatically achieve a preprogramed task of inserting a bolt in a flange hole and take over the human brain control as soon as it is triggered after the manipulator hand reach near the flange hole.

Also it has a vibration feedback when it cannot reach a solution because of any obstacle or high unexpected torque while screwing a nut to a bolt.

This invention provides two modes manual mode where the person / user wearing the helmet is totally in control of the operation , the user in manual mode wears the helmet and hold controllers to move arms , view the environment in 3d from the 3d stereo cameras and understand measurements coming from PRC software generated model. There is also an automatic mode which is the main function of the artificial intelligence software module, this module gathers all data coming from PRC, force and torque sensor to accomplish complete tasks autonomously upon already setuped missions.

[0015] (F) the photo realistic 3D cloud (PRC) system is used to recognize underwater features such as pipelines and subsea assets. Photo Realistic 3D Cloud (PRC) is a cutting- edge innovative technology to scan complete structures and pipeline sections underwater to create a 3D Cloud of millions of points presenting the as-built of any scanned object with a high accuracy more than 1/1000 (ex: lmm accuracy in a lm measurement).

The PRC help solution to pass exact dimensions to computer brain to perform accurate installation tasks like recognize a flange position and dimensions, therefore enable the MMAI to finish the task of installing all the other bolts and nuts on a flange

automatically.

[0016] It is worth mentioning that the camera feed is also continuously scanning the scene in 3D creasing an accurate dimension of all object which enable the operator to switch to autonomous operations for the manipulator ex: tie two flanges together by inserting bolts in all holes of the flange and insert nuts on the other side then tighten all nuts to a given torque. [0017] In the oil industry it is always challenging to visualize a full structure or pipeline section with precise as built dimensions due to the limited visibility underwater ( Average 4 to 6 meters ) and the limited accuracy of underwater positioning.

[0018] Photo Realistic 3D Cloud (PRC) is a cutting edge innovative technology to scan complete structures and pipeline sections underwater to create a 3D Cloud of millions of points presenting the as-built of any scanned object with a high accuracy more than 1/1000 (ex: 1mm accuracy in a lm measurement).

Using breakthrough advanced optical sensors and mathematical disciplines.

Equipment can be mounted on any size ROV (Remote Operated Vehicles subsea) or even a diver.

[0019] the photo realistic 3D cloud (PRC) system provides the below features:

1. Produce full set of as-found drawing base line for subsea structures within mm accuracy and precise 3D integrated model with all dimension.

2. Eliminate the need for verification as built visits, whenever new installation needed to eliminate the necessity for frequent Vessel visits to obtain physical measurement.

3. Faster, diver-less and more accurate approach than conventional way for collecting as-built/as-found information avoiding human error.

4. Precise 3D reference for all features position/orientation; convenient for future construction/maintenance (i.e. riser installation, flange/clamps fabrication).

5. Database can integrate within company Assets Integrity Data Management System.

6. In-depth Integrity assessment & life time extension.

7. PRC equipment can be mounted on ROV or divers.

8. (PRC) system of the present invention is the only way for user in manual mode to view the full object in operation, imagine the user is making an operation on the pipeline’s flange

Characterized in that: d Through helmet the operator is only viewing the flange in full view but through PRC the operator can see the full pipeline if surveyed before , flange’s dimensions , bolts and nuts dimensions and how many bolts installed for which pipe diameter .. etc

[0020] The present invention and in particular preferred embodiments according to the invention will be described with reference to the accompanying figures in more details as below: FIG. I schematically illustrates an embodiment of the enhanced reality underwater maintenance system including all the components of VRM System according to present invention which comprising VR helmet (1) and VR controllers (2), PRC cameras housing (3), Manipulator (4), ROV (remotely operated vehicle) (5), ZM camera (6).

[0021] FIG. 2 shows a particular embodiment of the top, front and side view of the used Manipulator (4) in the present invention, wherein the manipulators are arms equipped on ROV (5) to perform specific tasks. ROV manipulators can be as simple as 1 joint, up to 7 joints. Depends on the size, and the required tasks. Moreover, the manipulators can also be electric driven or hydraulic, and it can carry up to 380Kgs.

[0022] FIG. 3 shows a particular embodiment of the used ROV (remotely operated vehicle) (5) in the present invention and its top, front and side view. ROV (5) stands for remotely operated vehicle, it’s used to replace the divers where they can’t go, like deep water, extreme conditions, hazard areas. Its size varies from a hand carry size to a work class. Commonly the ROV is equipped with thrusters to enable it to do subsea maneuvering either by electric power or hydraulic pump driven by electric motor.

For medium and work class sizes, ROVs are equipped with 1 or 2 manipulators which can do basic or complicated tasks.

[0023] FIG. 4 shows a particular embodiment of the used PRC cameras housing (3) in the present invention and its top, front and side view. The used PRC cameras (3) are enclosed inside sealed aluminum enclosure to tolerate underwater pressure up to 3000m, Housing is equipped with dome as original camera dome can’t operate in water. All camera ports are equipped with underwater connectivity bulk heads to connect electricity and network to camera housing to be connected to rest of the system.

[0024] FIG. 5 shows a particular embodiment of the used Switch pod housing (7) in the present invention and its top, front and side view. The PRC solution require the existence of underwater network switch, this network switch do two main functions as below:

(1) It is used to synchronize the images taken between the two cameras at the exact precise time for processing images through PTP protocol Precision Time protocol.

(2) The other function is to convert the ethemet connectivity to fiber optic signal to be connected with ROV fiber optic cable to transfer taken images instantly to vessel top. Switch originally is not entitled to go under water, accordingly its enclosed into an aluminum housing / pod to be able to tolerate pressure up to 3000m under sea surface.

[0025] FIG. 6 shows a particular embod iment of the used ZM camera housing (6) in the present invention and its top, front and side view. Z-mini camera brings the best of virtual and augmented reality together. Using advanced depth sensing technology, this camera needed for virtual reality helmet to view underwater environment in 3D. This camera is enclosed into a housing to go underwater up to 3000m depth under sea surface.

[0026] FIG. 7 shows a particular embodiment of the used Pan and tilt unit for manipulator (8) in the present invention and its top, front and side view. Z-mini cameras are installed on pan and tilt unit, it allows cameras to move up and down, right and left in accordance with virtual reality controllers held by the operator while wearing the VR helmet (1).

[0027] FIG. 8 shows a particular embodiment of the Manipulator force and torque integration in the present invention and its top, front and side view, it shows how the force and torque sensor (9) is integrated inside the manipulator arm so the arm can sense used forces and torques to move objects or do specific operations.

[0028] The manipulator is equipped with torque and force sensor to read the force generated from different moves or fasten bolts with specific torque and precisely calculated movements from manipulator joints, such results are displayed as numbers on the VR image and transmitted as vibrations in the movement sensors held by the operator. FIG.9 schematically illustrates an embodiment of the photo realistic 3D cloud (PRC) system according to the present invention. FIG. 10 shows a particular embodiment of the used Force/Torque Sensor System, wherein every manipulator arm is equipped with 1 force and torque sensor.

BRIEF DESCRIPTION OF THE FIGURES:

[0010] The present invention and in particular preferred embodiments according to the invention will be described in more detail with reference to the accompanying figures.

FIG. 1 schematically illustrates an embodiment of the enhanced reality underwater maintenance system including all the components of VRM System according to present invention,

FIG. 2 shows a particular embodiment of the top, front and side view of the used Manipulator in the present invention.

FIG. 3 shows a particular embodiment of the used ROV (remotely operated vehicle) in the present invention and its top, front and side view.

FIG. 4 shows a particular embodiment of the used PRC cameras housing in the present invention and its top, front and side view.

FIG. 5 shows a particular embodiment of the used Switch pod housing in the present invention and its top, front and side view.

FIG. 6 shows a particular embodiment of the used ZM camera housing in the present invention and its top, front and side view.

FIG. 7 shows a particular embodiment of the used Pan and tilt unit for manipulator in the present invention and its top, front and side view.

FIG. 8 shows a particular embodiment of the Manipulator force and torque sensors in the present invention and its top, front and side view.

FIG.9 schematically illustrates an embodiment of the photo realistic 3D cloud (PRC) system according to the present invention.

FIG.10 shows a particular embodiment of the used Force