FIELD OF THE INVENTION

[0001] This invention relates to robotic manipulators and more particularly a device that uses robotic manipulators to provide haptic feedback to a user moving through a virtual environment.

BACKGROUND OF THE INVENTION

[0002] Generic industrial robotic manipulators are heavily utilized in manufacturing due to their versatility and relative low cost as compared to custom hardware. A vast majority of industrial robotic manipulators on the market are rigid devices capable of great precision and speed. Typically, this precision and speed make the industrial robotic manipulators unsuitable for human interaction due to the risk of injury to a person who is within the trajectory of the robotic manipulator.

SUMMARY OF THE INVENTION

[0003] The application is directed to a full body exoskeleton capable of providing haptic user feedback. Many embodiments are directed to a full body haptic exoskeleton device that includes a base tower which forms the main support for the upper and lower robotic manipulators. A first set of upper robotic manipulators may be mounted on opposing sides of the tower such that the first and second upper manipulators align with left and right arms of a user. The exoskeleton may also include a first set of lower robotic manipulators mounted on opposing sides of the tower such that the first and second lower manipulators align with left and right legs of a user. Each of the upper robotic manipulators may be outfitted with an arm coupling on an end effector of each of the first and second upper robotic manipulators for connecting to an arm of the user. Additionally, each of the lower robotic manipulators may be configured with a leg coupling on an end effector of the robotic manipulators for connecting to a leg of the user.

[0004] In other embodiments, the full body haptic exoskeleton includes a seating element, wherein the seating element supports the majority of the weight of the user. [0005] In yet other embodiments, the seating element is adjustable in position relative to the user such that the user position within the exoskeleton device is adequate for comfortable movement.

[0006] In still other embodiments, the upper and lower manipulators are configured to accommodate 90% of male human users.

[0007] In yet still other embodiments, each of the upper manipulators has an overall length of 760 mm and each of the lower manipulators has an overall length of 950mm.

[0008] In other embodiments, the full body haptic exoskeleton includes a control system, wherein the control system is in electronic communication with each of the upper and lower manipulators and wherein the control system is configured to receive input data and translate the input data into relative motion of the manipulators such that a non- colliding workspace overlap is maintained.

[0009] In yet other embodiments, the input data is received from a group consisting of motion sensors, force sensors, and a virtual reality control application.

[0010] In still other embodiments, the full body haptic exoskeleton includes an enclosure panel wherein the enclosure panel encloses the user within the exoskeleton device and may be used to better simulate a virtual reality environment.

[0011] In still yet other embodiments, the enclosure panel is in a fixed position and configured to have an opening in a portion thereof to allow a user to easily interact with the device.

[0012] In other embodiments, the full body haptic exoskeleton further includes a processing device having a processor and a memory component wherein the memory component is configured to receive and store instructions wherein the instructions may then be used by the processor.

[0013] In still other embodiments, the processor component may be a processor, microprocessor, controller, and/or a combination of processors, microprocessors, and controller.

[0014] In yet other embodiments, the memory component may be selected from either volatile memory or non-volatile memory.

[0015] In still yet other embodiments, each of the upper and lower manipulators further include end effectors wherein the couplings are disposed on the end effectors. [0016] In other embodiments, the arm couplings may be either stirrups or handles or a combination thereof.

[0017] In yet other embodiments, the end effectors on the lower manipulators further comprise slots into which a user can insert his or her feet.

[0018] Other embodiments include a method for configuring the various manipulators of the full body haptic exoskeleton device in which the position of each of the manipulators may be determined and thus positioned accordingly. Such embodiments may include having a base tower configured to or adaptable to receive a set of upper and lower robotic manipulators each having at least six degrees of freedom where each of the manipulators may be positioned on opposing sides of the base tower. An optimal user workspace may be determined for the exoskeleton device according to a majority of potential users. The workspace for the robotic manipulators may be determined based on the dimensions of the manipulators. Having a determined workspace for both the user and the robotic manipulators a workspace overlap can then be calculated. If the overlap of the two workspaces is such that a user could be adversely affected by the robotic manipulator movement, the overlap can be adjusted and further optimized such that the exoskeleton workspace encapsulates the optimal user workspace. Once the workspace overlap is configured such that the exoskeleton workspace encapsulates the user workspace the upper and lower manipulators can be mounted to the base tower such that while in use the overlap is maintained.

[0019] In other embodiments, determining the optimal user workspace and exoskeleton workspace further includes using three dimensional point cloud data wherein a plurality of corresponding points are generated and then compared with respect to the relative positioning.

[0020] In yet other embodiments, the optimizing the workspace overlap is performed using an iterative closest point algorithm.

[0021] In still other embodiments, optimizing the workspace overlap further includes determining a collision rejection variable in which any points within the workspace overlap that do not have a corresponding robotic pose or may interfere with the movement of the user, may be removed from the optimization calculation. [0022] Still other embodiments include a method for controlling a full body haptic exoskeleton which include having a user and a full body haptic exoskeleton device according the many embodiments. The full body haptic exoskeleton device may be configured to receive virtual reality information into a processing unit with respect to a desired virtual environment wherein the virtual environment may include multiple virtual objects within a virtual workspace. Furthermore, the virtual reality information may contain data with respect to the location and dimension of the virtual objects and the user. The device can translate the virtual objects and their respective positions into a virtual workspace. Additionally, as the virtual environment, including the objects, is determined, limitations on the movement of the upper and lower manipulators can be set to ensure the movement is within the workspace. The movement of each of the manipulators can then be calculated based on the workspace and the virtual environment. Finally, a controller device can control the movement of each of the manipulators of the exoskeleton device.

[0023] In other embodiments, calculating the movement of each of the upper and lower manipulators further includes; determining movement criteria. The movement criteria include determining if the movement of the manipulator will collide with the user, will exceed the operator joint limitations, and/or will collide with a virtual object. The movement of each manipulator may be revised to match the earlier determined criteria. Each of the manipulators may be controlled based on the calculated movement.

[0024] Additional embodiments and features are set forth in part in the description that follows, and in part will become apparent to those skilled in the art upon examination of the specification or may be learned by the practice of the disclosure. A further understanding of the nature and advantages of the present disclosure may be realized by reference to the remaining portions of the specification and the drawings, which forms a part of this disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 illustrates a front perspective view of an open full body haptic exoskeleton device in accordance with an embodiment of the invention.

[0026] Figure 2 illustrates a front perspective view of a controlled environment full body haptic exoskeleton device in accordance with an embodiment of the invention.

[0027] Figure 3 illustrates graphs showing the results of an iterative closest point calculation showing the movement of the workspaces to their closes overlapped state of robotic manipulators in a haptic device in accordance with an embodiment of the invention.

[0028] Figure 4 illustrates a showing the overlap workspace of the left human arm with a robotic arm workspace in a haptic device in accordance with an embodiment of the invention.

[0029] Figure 5 illustrates a processing system that controls the manipulators in accordance with an embodiment of the invention.

[0030] Figure 6 illustrates a flow diagram of a process performed by the processing system to control the movement of the manipulators in accordance with an embodiment of the invention.

DETAILED DISCLOSURE OF THE INVENTION

[0031] Turning now to the drawings, full body haptic exoskeleton systems in accordance with various embodiments of the invention are disclosed. In accordance with some embodiments of the invention, four industrial robotic manipulators are used to create a full body haptic device that allows for haptic interactions with both arms and both legs of a user. In accordance with some embodiments of the invention, impedance control of an industrial robotic manipulator can be used to make the manipulator act as a follower of and resistor to human applied input. One such use is the provision of haptic feedback to a user. In order to provide haptic feedback, a system can advantageously use the rigidity of these industrial robotic manipulators by attaching an end effector of an industrial robotic manipulator to a limb or limbs of a human operator. The manipulator is then used to resist movement of the limb by the human operator in the real world to reflect objects or boundaries in a virtual world. While it is unorthodox to pair a human with a rigid industrial robotic manipulator, this type of system allows for a very strong and stiff reaction when the user interacts with a solid and/or immovable virtual object such as a wall. Other types of haptic systems use compliant robotic systems that are designed to interact with humans by mechanically flexing. However, these compliant systems result in less immersive experiences for the users.

[0032] Thus, a true full body virtual experience can be created by the device. In accordance with many embodiments, the robotic manipulators that interact with the limbs of a user are positioned in the device with respect to the human operator via rigorous calculations to achieve high overlapping workspace coverage and matching manipulability of the robotic manipulators to the maneuverability of human limbs. A more complete discussion of a full body haptic exoskeleton system in accordance with various embodiments of the invention follows.

Full Body Haptic Exoskeleton Devices

[0033] A full body haptic exoskeleton device in accordance with an embodiment of the invention is shown in Figure 1 . A full body haptic exoskeleton device 100 in accordance with an embodiment of the invention includes four industrial robotic manipulators 101 -104 connected in parallel with a human operator. The lower two robotic manipulators 102, 104 attach to the lower extremities of human operator in the shown embodiment. In accordance with some embodiments, the connection is at the ankles of the user. In accordance with several embodiments, the connection may be at the foot and in accordance many embodiments, the connection may be to the lower leg of the user.

[0034] The upper two robotic manipulators 101 , 103 are attached to the wrists of an operator in the shown embodiments. In accordance with several embodiments, the connection may be to the hand of the operator and in still further embodiments, the connection may be to the forearm of the operator.

[0035] The connections to the upper and lower extremities of the operator may be provided via one of two configurations in accordance with some embodiments. A stirrup/handle tooltip can be located on the end effector of each of the upper robotic manipulators that can be grasped by an operator. In addition, the end effector of the lower robotic manipulators can have slots into which a user can insert his or her feet. This configuration allows for quick mounting and dismounting of the operator in the device. A second human-device interface configuration has wrist and ankle mounted clutches. The wrist and ankle clutches allow for free human hand and ankle orientation and break away from the robotic arms should a force pass a pre-designated threshold value.

[0036] In the illustrated embodiment, the four robotic manipulators 101 -104 are mounted on a command tower 1 10 that houses the controllers as well as a seat 105 for the operator. The seat 105 is optional and is designed to move in the vertical direction to accommodate persons of varying heights and may be configured to improve the comfort of the user when operating the exoskeleton device. The seat 105 may also be driven during operation to enhance the haptic experience in accordance with many embodiments. The seat 105 reduces the load that the robotic manipulators need to carry by supporting most of the weight of the operator. In accordance with several embodiments, a harness may be provided to support the weight of the operator. The entire device can be built as open environments as shown in Figure 1 or as an enclosed, controlled environment as shown in Figure 2. In device 200, environmental control systems may control airflow and moisture in enclosure 205 to enhance the haptic experience. In accordance with an embodiment the enclosure 205 may be integrated with the command tower 1 10 and may be permanently fixed with an opening 210 to allow for the user to enter and exit the exoskeleton device 100. In other embodiments the enclosure may include a moveable panel that may be moved out of the way of the exoskeleton device 100 to better accommodate the user while positioning themselves within the device 100.

Optimal Manipulator Orientation

[0037] In accordance with some embodiments, the optimal placement of the mounting for each robotic manipulator with respect to a coupled human limb considers several criteria that should ideally be met for each robotic manipulator. In accordance with many embodiments, a mounting placement determination method that optimizes a position of a robotic manipulator with a weighted function based on human-robot workspace overlap, manipulability similarity, collision rejection, and operation location probability is used to determine the mount position of each robotic manipulator. Human- robot workspace overlap can be the most important factor in the weighted optimization to reduce the likelihood that the device restricts the range of movements that an operator might want to perform in the device. Furthermore, achieving maximum overlap in the human-robot workspace can increase the comfort level of the human user of the device.

[0038] To evaluate human-robot workspace overlap, methods in accordance with many embodiments of the invention generate a workspace of the robotic manipulator as point cloud data and compare the workspace to a simulated human arm workspace based on the joint limits of a healthy human subject. In accordance with many embodiments, the configuration and placement of the manipulators to accommodate 90% of human male users and thus accommodate the majority of users. Comparison may be performed using an iterative closest point algorithm in accordance with many embodiments to find a volumetric coverage of the human limb's workspace (Voverlap/VHuman Total). The Iterative Closest Point (ICP) algorithm can compare point data from two different data sets and utilizes an iterative process to create an overlap of the two data sets within a defined error limit. In accordance with many embodiments, the two data sets may be the point data sets from the human and robotic arm workspaces. Using the Euclidean distance formula, the ICP algorithm may determine the distance between points. Subsequently, the algorithm moves or adjusts, calculates distances, and readjusts iteratively until a specified error is met.

[0039] The ICP algorithm may be utilized in a haptic exoskeleton device within a virtual reality environment to improve the response of the exoskeleton within the human workspace. Furthermore, the ICP method can be useful in preventing the movement of the exoskeleton manipulators from exceeding the range of motion of the user thereby reducing the risk to the user.

[0040] A graph showing a sample iterative closest point calculation is shown in Figure 3. In Figure 3, the first plot 300 shows the work space of an upper robotic manipulator 320 and the work space of a human arm 325. The second plot 305 shows the result an iterative closest point comparison wherein the workspace of the human arm 325 is enclosed within the workspace of an upper robotic manipulator 320. Graph 310 shows the convergence of an iterative closest point comparison.

[0041] A visualization of a mounting position of a left upper robotic manipulator with respect to a left arm of a user in accordance with an embodiment of the invention is shown in Figure 4. In Figure 4, a diagram shows the overlap workspace of the left human arm with a robotic arm workspace in accordance with an embodiment of the invention. The overlap workspace between the human arm and the robotic manipulator, according to many embodiments, may be determined best by using the iterative point method described previously. The reachable workspace of the left human arm is shown in green, the unreachable workspace of the arm in red, and total robotic manipulator workspace in blue. In order to impose no restriction in 3D space on the user, the selected manipulators are typically chosen to have at least six degrees of freedom. When six degrees of freedom are chosen, this is less than the human seven degrees of freedom per limb (from shoulder to wrist and hip to ankle) but the redundancy is not needed in a mechanical haptic device. The robotic manipulators can also be chosen to have a reach such that there is a minimum of wasted workspace space beyond the workspace of the human limbs. For a 90-percentile male this is 760mm reach for upper manipulators and 950 mm reach for lower manipulators. Payloads are also important to provide adequate haptic feedback forces/torques, which for the system could be a minimum amount to support the human leg and arm without any human assistance. This would mean a minimum of 4kg for the upper limbs and 12kg for the lower limbs. Large payloads may also be selected to increase the feedback magnitude capabilities. In many embodiments, the robot joint centers do not coincide with the human joint locations.

[0042] In accordance with a number of embodiments, manipulability is also considered in selection of the placement of the mount of each manipulator to ensure the robotic manipulator can follow the human operator with relative ease. To do so, placement determination methods in accordance with some embodiment determine mounting locations that satisfy the workspace overlap criteria. For each of these locations, manipulability of the robotic manipulator is calculated at numerous workspace points and compared to the manipulability of the corresponding human limb at the same points. By using the human limb manipulability as a reference, a score is assigned to how closely manipulability of the robotic manipulator matches for all possible motions of the corresponding limb. The score is used in the final optimization to select the best mounting location.

[0043] While workspace overlap and manipulability are the dominant factors driving optimal placement location for mounting each of the robotic manipulators, collision rejection and workspace location probability may also be used as filters in placement determination methods in accordance with many embodiments of the invention to remove any configurations that may be unsuitable physically. In the determination method in accordance with some of these embodiments, collision rejection is evaluated by examining the overlapping workspace of the human-robot system (described above) and removing any points that do not have suitable robotic arm poses and do not interfere with the movement of the corresponding limb of the operator. Workspace data may also be evaluated by comparing the workspace to the probability of human activity in specific regimes of the workspace of the corresponding limb of an operator. Areas where the operator is less likely to move or place the limb are less critical than areas of high activity for the limb. For example, it is more important to have unrestricted human arm movement (both position and velocity based) in the area directly in front of the torso than it is behind the back of the operator where the operator seldom reaches.

Control of the Manipulators

[0044] In accordance with some embodiments of the invention, movement of the robotic manipulators is controlled to provide haptic feedback for a virtual reality environment. A control system receives information about the virtual reality environment and controls the manipulators based upon the pose of the operator within the virtual reality environment. For example, the virtual reality environment may include a wall and the full body haptic device prevents the user from adopting a pose that would correspond to movement through the wall in the virtual reality environment. To do so, the control system may include a processing system that executes software instructions for a control application to control the robotic manipulators of the device. A processing system that is part of the control system in accordance with some embodiments of the invention is shown in Figure 5. In accordance with several embodiments, the processing system may be housed within the device. In accordance with a number of embodiments, the processing system may be outside of the device and communicates with controllers inside the device via a hard-wired or wireless connection. One skilled in the art will recognize that a particular processing system may include other components that are omitted for brevity without departing from this invention.

[0045] The processing device 500, shown in Figure 5, includes a processor 505, a non-volatile memory 510, and a volatile memory 515. The processor 505 is a processor, microprocessor, controller, or a combination of processors, microprocessor, and/or controllers that performs instructions stored in the volatile memory 515 or non-volatile memory 510 to manipulate data stored in the memory. The non-volatile memory 510 can store the processor instructions utilized to configure the processing system 500 to perform processes including processes in accordance with embodiments of the invention and/or data for the processes being utilized. In other embodiments, the processing system software and/or firmware can be stored in any of a variety of non- transient computer readable media appropriate to a specific application. A network interface is a device that allows processing system 500 to transmit and receive data over network based upon the instructions performed by processor 505. Although a processing system 500 is illustrated in FIG. 5, any of a variety of processing systems in the various devices can configured to provide the methods and systems in accordance with embodiments of the invention can be utilized.

[0046] A control application executed by the processing system in accordance with some of these embodiments maps a virtual reality environment to a workspace environment of the device. Based on the mapping of the virtual reality environment to the workspace, the parameters for movements of the manipulators within the workspace are set such the manipulators provide haptic feedback for movement by the user within the virtual reality environment. A process for controlling the robotic manipulators of a full body haptic device in accordance with an embodiment of the invention is shown in Figure 6.

[0047] In process 600, virtual reality environment information is received (605). In accordance with some embodiments, the virtual reality information includes information about the location and/or dimensions of objects in the environment. The virtual reality information may also include position and orientation information for the user within the virtual environment.

[0048] The position and dimension of objects and other structures in the virtual reality information are then translated from the virtual reality space to coordinates within the workspace of the manipulators of the full body haptic device (610). The workspace coordinates of the positions and dimension of objects and other structures in the virtual reality environment are then used to set limits on the movements of each of the robotic manipulators (615). This process may then be periodically repeated as new virtual reality information is received by the processing system.

[0049] The process 600 then begins to receive input of operator's movements from the sensors mounted on the manipulators indicating desired movements by the operator in the virtual reality world (620). A movement for each manipulator to emulate the user's movement in the virtual reality world is then determined from the received input from the sensors (625). The movement for each manipulator is then analyzed to determine whether, the movement will cause a collision with operator (630). If a possible collision is detected, the movement of each manipulator is revised (645) then revised movements are retested. The movements are also tested to determine whether the movement will violate an operator joint limit (635). If a possible violation of a joint limit is detected, the movement of each manipulator is revised (645) then revised movements are retested. The movements are further tested to see if a collision with a VR object occurs (640). If a possible collision is detected, the movement of each manipulator is revised (645) then revised movements are retested. In many embodiments, the control scheme is dominated by admittance control allowing the operator to move freely with no resistance. Upon coming in contact with a virtual object, a combination of admittance control and impedance control can be used to simulate a virtual stiffness of the object. This can range from completely rigid (thus not allowing any penetration by the user's limbs for a virtual wall or similar object) to semi-compliant (such as for a virtual fluid).

[0050] If the movement is acceptable, the movement for each manipulator is sent to the manipulators (650) and the process repeats until an end signal is detected (655).

[0051] Although a process for controlling the manipulators in a full body haptic device in accordance with an embodiment of the invention is described above with reference to Figure 6, other processes for controlling the manipulators that add steps to, remove steps from, or combine steps of the described process may be used in accordance with various other embodiments of the invention.

[0052] Although specific full body haptic devices and processes for controlling the manipulators in a full body haptic device are discussed above, many different systems and methods can be implemented in accordance with various embodiments of the invention. It is therefore to be understood that the present invention may be practiced in ways other than specifically described, without departing from the scope and spirit of the present invention. Thus, embodiments of the present invention should be considered in all respects as illustrative and not restrictive. Accordingly, the scope of the invention should be determined not by the embodiments illustrated, but by the appended claims and their equivalents.