System

Related Application

[0001] This application claims the benefit of U.S. Provisional Application No. 62/646,063, filed on 21 March 2018; the contents of which are incorporated herein by reference.

Field of the Invention

[0002] The embodiments of the invention generally relate to methods for haptic feedback in a virtual reality system.

Background

[0003] Virtual reality systems generally allow for computer-generated interactive experiences that may occur within a simulated environment. Existing VR technology commonly uses headsets or multi-projected environments, sometimes in combination with physical environments, to generate realistic images, sounds, and other sensations that simulate a user’s physical presence in a virtual or imaginary environment.

Summary

[0004] In one or more embodiments of the present disclosure, a system for providing haptic feedback is provided. Embodiments may include a wearable virtual reality feedback device. The device may include a wearable glove having a plurality of piezo electric devices. Each piezo-electric device may be located on a different portion of the wearable glove and may be configured to provide haptic feedback to a user.

[0005] One or more of the following features may be included. In some embodiments, each of the plurality of piezo-electric devices may be configured to simulate a force impulse response of a real world device. The system may include a storage device configured to store at least one of a force response over distance curve or a response over time curve corresponding to a real-world device. The system may also include a graphical user interface configured to allow for reuse of the stored force response over distance curve to mimic a real world haptic interaction in a virtual reality application. The force response over distance curve may be based upon, at least in part, at least one of force response over distance data or force response over time data. The real world device may be a user-selectable device. A plurality of force response over distance curves may be stored and each curve may correspond to a different one of the real world devices. The haptic feedback may be biased or scaled using electromyography. The electromyography data may be obtained from the user and provided as biofeedback. The electromyography data may be received at the wearable device from a wearable biological monitor. The user-selectable device may include one or more of a switch, knob, button, toggle trigger, pushbutton.

[0006] In one or more embodiments of the present disclosure, a virtual reality feedback method is provided. The method may include providing a wearable glove and providing haptic feedback to a user via a plurality of piezo-electric devices, wherein each piezo-electric device is located on a different portion of the wearable glove.

[0007] One or more of the following features may be included. In some embodiments, the method may include simulating a force impulse response of a real world device using each of the plurality of piezo-electric devices. The method may further include storing at least one of a force response over distance curve or a response over time curve corresponding to a real-world device at a storage device. The method may also include using a graphical user interface configured to allow for reuse of the stored force response over distance curve to mimic a real world haptic interaction in a virtual reality application. The force response over distance curve may be based upon, at least in part, at least one of force response over distance data or force response over time data. The real world device may be a user-selectable device. A plurality of force response over distance curves may be stored and wherein each curve corresponds to a different one of the real world devices. The haptic feedback may be biased or scaled using electromyography. The electromyography data may be obtained from the user and provided as biofeedback.

[0008] Additional features and advantages of embodiments of the present disclosure will be set forth in the description which follows, and in part will be apparent from the description, or may be learned by practice of embodiments of the present disclosure. The objectives and other advantages of the embodiments of the present disclosure may be realized and attained by the structure particularly pointed out in the written description and claims hereof as well as the appended drawings.

[0009] It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory and are intended to provide further explanation of embodiments of the invention as claimed.

Brief Description of the Drawings

[0010] The present invention is illustrated by way of example, and not limitation, in the figures of the accompanying drawings in which like references indicate similar elements and in which:

[0011] FIG. 1 illustrates a block diagram of an exemplary system for VR based positional tracking in accordance with embodiments of the present disclosure;

[0012] FIG. 2 illustrates a flowchart showing operations consistent with embodiments of the present disclosure;

[0013] FIG. 3 illustrates an example VR system including a VR glove consistent with embodiments of the present disclosure;

[0014] FIG. 4 illustrates an example VR system including a VR glove consistent with embodiments of the present disclosure;

[0015] FIG. 5 illustrates an example placement location for inverse kinematics driven hand control consistent with embodiments of the present disclosure;

[0016] FIG. 6 illustrates an example high resolution 3D skeletal bone structure for inverse kinematic driven hand control;

[0017] FIG. 7 illustrates a schematic of a sensor network interface consistent with embodiments of the present disclosure;

[0018] FIG. 8 illustrates a forearm/wrist positional sensor consistent with embodiments of the present disclosure;

[0019] FIG. 9 illustrates an example of real-time position tracking and continuous calibration consistent with embodiments of the present disclosure;

[0020] FIG. 10 illustrates an example of a generic sensor connection to I2C bus consistent with embodiments of the present disclosure;

[0021] FIG. 11 illustrates an example network connection consistent with embodiments of the present disclosure;

[0022] FIG. 12 illustrates a network initialization controller consistent with embodiments of the present disclosure;

[0023] FIG. 13 illustrates a network initialization module consistent with embodiments of the present disclosure;

[0024] FIG. 14 illustrates example protocol states consistent with embodiments of the present disclosure;

[0025] FIG. 15 illustrates an example of hardware neighbor bus signaling consistent with embodiments of the present disclosure;

[0026] FIG. 16 illustrates an example asynchronous communication using the I2C data line consistent with embodiments of the present disclosure;

[0027] FIG. 17 illustrates an example electrical schematic for position tracker board consistent with embodiments of the present disclosure;

[0028] FIG. 18 illustrates an example physical schematic for position tracker board consistent with embodiments of the present disclosure;

[0029] FIG. 19 illustrates an example position tracker board consistent with embodiments of the present disclosure;

[0030] FIG. 20 illustrates a plot showing keyboard keypress force over distance consistent with embodiments of the present disclosure;

[0031] FIG. 21 illustrates a plot showing tactile switch force over distance consistent with embodiments of the present disclosure;

[0032] FIG. 22 illustrates an example of EMG reticle simulation with support for EMG threshold training to identify applicable forces and interactions during virtual interaction consistent with embodiments of the present disclosure;

[0033] FIG. 23 illustrates an example depicting an alpha numeric keypad interaction test with EMG data recorded from an electrical output measurement device consistent with embodiments of the present disclosure;

[0034] FIG. 24 illustrates an example depicting a multi-functional display (MFD) interaction test with associated EMG data consistent with embodiments of the present disclosure;

[0035] FIG. 25 illustrates an example depicting a cyclic interaction test with associated EMG used to control vibrational response consistent with embodiments of the present disclosure;

[0036] FIG. 26 depicts an example showing EMG muscle activity during interaction with digital cyclic flight stick consistent with embodiments of the present disclosure; and

[0037] FIG. 27 shows an example showing haptic device paired with an electrical output measurement device armband to provide biofeedback with proportional vibration based on EMG muscle activity consistent with embodiments of the present disclosure.

Detailed Description

[0038] Reference will now be made in detail to the embodiments of the present disclosure, examples of which are illustrated in the accompanying drawings. The present disclosure may, however, be embodied in many different forms and should not be construed as being limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the concept of the disclosure to those skilled in the art. [0039] As will be appreciated by one skilled in the art, the present disclosure may be embodied as a method, system, or computer program product. Accordingly, the present disclosure may take the form of an entirely hardware embodiment, an entirely software embodiment (including firmware, resident software, micro-code, etc.) or an embodiment combining software and hardware aspects that may all generally be referred to herein as a“circuit,”“module” or“system.” Furthermore, the present disclosure may take the form of a computer program product on a computer-usable storage medium having computer-usable program code embodied in the medium.

[0040] As used in any embodiment described herein,“circuitry” may include, for example, singly or in any combination, hardwired circuitry, programmable circuitry, state machine circuitry, and/or firmware that stores instructions executed by programmable circuitry. It should be understood at the outset that any of the operations and/or operative components described in any embodiment herein may be implemented in software, firmware, hardwired circuitry and/or any combination thereof.

[0041] Any suitable computer usable or computer readable medium may be utilized. The computer readable medium may be a computer readable signal medium or a computer readable storage medium. A computer-usable, or computer-readable, storage medium (including a storage device associated with a computing device or client electronic device) may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer-readable medium may include the following: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device. In the context of this document, a computer-usable, or computer-readable, storage medium may be any tangible medium that can contain, or store a program for use by or in connection with the instruction execution system, apparatus, or device.

[0042] A computer readable signal medium may include a propagated data signal with computer readable program coded embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated signal may take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device.

[0043] Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0044] Computer program code for carrying out operations of the present invention may be written in an object oriented programming language such as Java, Smalltalk, C++ or the like. However, the computer program code for carrying out operations of the present invention may also be written in conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user’s computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer may be connected to the user’ s computer through a local area network (LAN) or a wide area network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0045] The present disclosure is described below with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks.

[0046] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer- readable memory produce an article of manufacture including instructions which implement the function/act specified in the flowchart and/or block diagram block or blocks.

[0047] The computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks.

[0048] Referring to FIG. 1, there is shown a virtual reality haptic feedback process 10 that may reside on and may be executed by server computer 12, which may be connected to network 14 (e.g., the Internet or a local area network). Examples of server computer 12 may include, but are not limited to: a personal computer, a server computer, a series of server computers, a mini computer, and a mainframe computer. Server computer 12 may be a web server (or a series of servers) running a network operating system, examples of which may include but are not limited to: Microsoft® Windows® Server; Novell® NetWare®; or Red Hat® Linux®, for example. (Microsoft and Windows are registered trademarks of Microsoft Corporation in the United States, other countries or both; Novell and NetWare are registered trademarks of Novell Corporation in the United States, other countries or both; Red Hat is a registered trademark of Red Hat Corporation in the United States, other countries or both; and Linux is a registered trademark of Linus Torvalds in the United States, other countries or both.) Additionally / alternatively, virtual reality haptic feedback process 10 may reside on and be executed, in whole or in part, by a client electronic device, such as a personal computer, notebook computer, personal digital assistant, or the like.

[0049] The instruction sets and subroutines of virtual reality haptic feedback process 10, which may include one or more software modules, and which may be stored on storage device 16 coupled to server computer 12, may be executed by one or more processors (not shown) and one or more memory modules (not shown) incorporated into server computer 12. Storage device 16 may include but is not limited to: a hard disk drive; a solid state drive, a tape drive; an optical drive; a RAID array; a random access memory (RAM); and a read-only memory (ROM). Storage device 16 may include various types of files and file types.

[0050] Server computer 12 may execute a web server application, examples of which may include but are not limited to: Microsoft IIS, Novell Webserver™, or Apache® Webserver, that allows for HTTP (i.e., HyperText Transfer Protocol) access to server computer 12 via network 14 (Webserver is a trademark of Novell Corporation in the United States, other countries, or both; and Apache is a registered trademark of Apache Software Foundation in the United States, other countries, or both). Network 14 may be connected to one or more secondary networks (e.g., network 18), examples of which may include but are not limited to: a local area network; a wide area network; or an intranet, for example.

[0051] Server computer 12 may execute an electronic design automation (EDA) application (e.g., EDA application 20), examples of which may include, but are not limited to those available from the assignee of the present application. EDA application 20 may interact with one or more EDA client applications (e.g., EDA client applications 22, 24, 26, 28) for electronic design optimization.

[0052] Virtual reality haptic feedback process 10 may be a stand alone application, or may be an applet / application / script that may interact with and/or be executed within EDA application 20. In addition / as an alternative to being a server-side process, virtual reality haptic feedback process 10 may be a client-side process (not shown) that may reside on a client electronic device (described below) and may interact with an EDA client application (e.g., one or more of EDA client applications 22, 24, 26, 28). Further, virtual reality haptic feedback process 10 may be a hybrid server-side / client- side process that may interact with EDA application 20 and an EDA client application (e.g., one or more of client applications 22, 24, 26, 28). As such, virtual reality haptic feedback process 10 may reside, in whole, or in part, on server computer 12 and/or one or more client electronic devices.

[0053] The instruction sets and subroutines of EDA application 20, which may be stored on storage device 16 coupled to server computer 12 may be executed by one or more processors (not shown) and one or more memory modules (not shown) incorporated into server computer 12.

[0054] The instruction sets and subroutines of EDA client applications 22, 24, 26, 28, which may be stored on storage devices 30, 32, 34, 36 (respectively) coupled to client electronic devices 38, 40, 42, 44 (respectively), may be executed by one or more processors (not shown) and one or more memory modules (not shown) incorporated into client electronic devices 38, 40, 42, 44 (respectively). Storage devices 30, 32, 34, 36 may include but are not limited to: hard disk drives; solid state drives, tape drives; optical drives; RAID arrays; random access memories (RAM); read-only memories (ROM), compact flash (CF) storage devices, secure digital (SD) storage devices, and a memory stick storage devices. Examples of client electronic devices 38, 40, 42, 44 may include, but are not limited to, personal computer 38, laptop computer 40, mobile computing device 42 (such as a smart phone, netbook, or the like), notebook computer 44, for example. Using client applications 22, 24, 26, 28, users 46, 48, 50, 52 may access EDA application 20 and may allow users to e.g., utilize virtual reality haptic feedback process 10.

[0055] Users 46, 48, 50, 52 may access EDA application 20 directly through the device on which the client application (e.g., client applications 22, 24, 26, 28) is executed, namely client electronic devices 38, 40, 42, 44, for example. Users 46, 48, 50, 52 may access EDA application 20 directly through network 14 or through secondary network 18. Further, server computer 12 (i.e., the computer that executes EDA application 20) may be connected to network 14 through secondary network 18, as illustrated with phantom link line 54.

[0056] The various client electronic devices may be directly or indirectly coupled to network 14 (or network 18). For example, personal computer 38 is shown directly coupled to network 14 via a hardwired network connection. Further, notebook computer 44 is shown directly coupled to network 18 via a hardwired network connection. Laptop computer 40 is shown wirelessly coupled to network 14 via wireless communication channel 66 established between laptop computer 40 and wireless access point (i.e., WAP) 68, which is shown directly coupled to network 14. WAP 68 may be, for example, an IEEE 802.1 la, 802.1 lb, 802.1 lg, Wi-Fi, and/or Bluetooth device that is capable of establishing wireless communication channel 66 between laptop computer 40 and WAP 68. Mobile computing device 42 is shown wirelessly coupled to network 14 via wireless communication channel 70 established between mobile computing device 42 and cellular network / bridge 72, which is shown directly coupled to network 14.

[0057] As is known in the art, all of the IEEE 802.1 lx specifications may use Ethernet protocol and carrier sense multiple access with collision avoidance (i.e., CSMA/CA) for path sharing. The various 802.1 lx specifications may use phase-shift keying (i.e., PSK) modulation or complementary code keying (i.e., CCK) modulation, for example. As is known in the art, Bluetooth is a telecommunications industry specification that allows e.g., mobile phones, computers, and personal digital assistants to be interconnected using a short-range wireless connection.

[0058] Client electronic devices 38, 40, 42, 44 may each execute an operating system, examples of which may include but are not limited to Microsoft Windows, Microsoft Windows CE®, Red Hat Linux, or other suitable operating system. (Windows CE is a registered trademark of Microsoft Corporation in the United States, other countries, or both.).

[0059] Referring now to FIG. 2, an exemplary flowchart 200 depicting operations consistent with VR haptic feedback process 10. The method may include providing (202) a wearable glove and providing (204) haptic feedback to a user via a plurality of piezo-electric devices, wherein each piezo-electric device is located on a different portion of the wearable glove. It should be noted that electrical output measurement devices may be used to bias the system (as is discussed in further detail below) and the process may be used with ERMs (eccentric rotational mass) or any other haptic feedback method.

[0060] Referring now to FIGS. 3-19, embodiments of virtual reality haptic feedback process 10 are provided. Embodiments of the present disclosure are designed to be both modular and composable, allowing individual components to be mixed and matched depending on the needs of the training system. Embodiments included herein are capable of procedurally generating highly-detailed virtual environments in real time, making it suitable for near instant transmission.

[0061] In some embodiments, virtual reality haptic feedback process 10 may include a real-time haptics mapping engine and toolset for MR/VR training simulations. In some embodiments, a point-of-need delivery service is included that may be used for simulated, high-fidelity, training environments through the use of MR/VR and Haptic surrogates. In some embodiments, a cross-platform, high-performance, high-fidelity, scalable presentation client is included that may be capable of supporting real-time haptics mapping in simulated training environments.

[0062] In some embodiments, virtual reality haptic feedback process 10 may include a positional tracking, synthetic proprioception and haptic feedback system. Embodiments included herein may provide for improved 3D simulation training availability and effectiveness, significantly reduced capital costs associated with 3D motion simulation training systems, significantly improved motion tracking for more direct representation of physical interactions, and significant improvement to current state of the art initialization, calibration and setup.

[0063] Referring now to FIGS. 3-4, examples of a virtual reality positional tracking system where users may interact with platform components in a virtualized AH64D cockpit while receiving critical vibrotactile feedback. Accordingly, embodiments of virtual reality haptic feedback process 10 may incorporate a series of interrelated components with silicone rings that provide finger-level tactile feedback. Initial development provided finger-level positional tracking through infrared sensors as well as through inertial measurement unit (IMU) sensors that may be configured to track arm, hand and finger positions even while the user’s hands are not in view of the head- mounted display (HMD).

[0064] Embodiments included herein are directed towards VR systems and devices such as the glove discussed in further detail below. Embodiments of the VR glove described herein include a mature, robust and modular haptic feedback and finger- level positional tracking system (both hardware and software) that may be integrated with any selected VR or AR system with gaming or simulation (training intent).

[0065] In some embodiments, virtual reality haptic feedback process 10 may utilize a paired Inertial Measurement Unit (IMU) and microcontroller unit (MCU) set to perform real time positional tracking. This paired set may allow for custom firmware at the point of detection that includes IMU interface code, filtering and all of the data handling required to complete a functioning positional tracking system.

[0066] In some embodiments, the IMU may include a MEMS 10DOF motion sensor that includes a nine axis, gyroscope, accelerometer and compass as well as a thermal sensor used for calibration. The MPU may include a small QFN chipset in a 3x3x1 mm package that incorporates two dies within the chipset one for the gyro and accelerometer and the other for the compass.

[0067] In some embodiments, the MCU and IMU may be included on a single printed circuit board (PCB) that is small enough to locate on one segment of each digit of the hand. By design this allows three IMU/MCU sets per digit for a total of 15 per hand. The following diagram (FIG. 5) shows the typical location of the sensors. These positions correlate with the ideal location for the Inverse Kinematic (IK) calculation positions. Inverse Kinematics, as used herein, relates to the mathematical process of recovering the movements of the hand/fingers in the virtual world from the sensor input of the motion sensor. In some embodiments, inverse kinematics makes use of the kinematics equations to determine the joint parameters that provide a desired position for each of the user’s end-effectors. Once the user’s hands/fingers motions are determined from the IMU sensor tracker (or other external positional sensor), they can be used to determine where to connect the user’s hands/fingers to the world.

[0068] FIG. 5 depicts an example showing the marker placement, hand local reference system and finger joint angles. Markers position. Mi: head of the metacarpal bone of finger i (i = 1-5); Pi: head of proximal phalanx of finger i (i = 1-5); Di: head of distal phalanx of the thumb (i = 1) and head of middle phalanx of long fingers (i = 2-5); SU: styloid process of ulna; SR: styloid process of radius. Local reference system XYZ. The origin is in correspondence of the marker M2. Vectors (M2-M5) and (M2 - SR) define the metacarpal plane of the hand (grey triangle). Z-axis is normal to the metacarpal plane pointing palmarly, Y-axis has the direction of vector (M2 - SR) pointing distally, while X-axis is calculated as the cross-product of Y and Z-axis, pointing radially. Joint angles in transverse plane YZ (a) and in sagittal plane XY (b) of the hand. MCPJi: metacarpophalangeal joint flexion angle of finger i (i = 1- 5); IPJi: proximal interphalangeal joint flexion angle of finger i (i = 1-5); TAB: thumb abduction angle. MCPJi (i = 2-5) is defined as the angle between Y-axis and the projection of the vector (Pi - Mi) on the YZ plane; IPJi (i = 2-5) is the angle between the projections of vectors (Di-Pi) and (Pi-Mi) on the YZ plane. TAB is the angle between the vector (Pl - Ml) and the XY plane. MCPJ1 is the angle between X-axis and the projection of vector (Pl - Ml) on the XY plane. IPJ1 is the angle between vectors (Dl - Pl) and (Pl - Ml).

[0069] Following this schema, embodiments included herein have adopted a high resolution skeletal bone structure within our bipedal (human) models which corresponds to the marker placement reference. FIG. 6 depicts a high resolution 3D skeletal bone structure for Inverse Kinematic (IK) driven hand control. With an Inverse Kinematics (IK) approach, embodiments included herein may be configured to use the motion tracking IMU sensor (or other positional data) to move an IK handle on the fingertip to pose the entire joint chain (a joint chain that has an IK handle is called an IK chain). In this way, as the user moves their hands/fingers the IK solver automatically rotates all the joints in the IK chain. The IK solver may be used to calculate the rotations of all the joints in the IK chain as the user moves their hands/fingers. The high level per sensor design is shown in FIG. 7.

[0070] As shown in FIG. 7, the motion sensor may be a unified tracking, haptics feedback, communications and computing/processing device. The MCU may be placed and/or collocated with the IMU/MPU and connectivity between sensors utilizes an enhanced protocol while connectivity to subordinate devices such as the haptic feedback, or IMU/MPU can be made with standard electronics connectivity methods such as I2C and SPI. An example of the forearm/wrist controller and individual sensor components is provided in FIG. 8.

[0071] In some embodiments, the present disclosure may provide continuous calibration through pulse encoded LED illuminator tracking. While the use of MCU/IMU based motion tracking does allow for high speed tracking of points and skeletal system through the use of properly positioned points there are several aspects of the design that are not ideal. The first aspect is the initialization calibration at startup that needs to be done to correlate the IMU points to a skeletal system. Simple "games" have been developed that increase in fidelity and refinement that allow the user to gradually calibrate the system. While this progressive calibration may be effective and works well for most users, it can be time consuming. The second issue with IMU based image trackers is related to drift of the location either through cumulative delta position offset errors such as a floating point rounding or significant digits error, or an error due to external inputs such as drift caused by changing thermals of the chipset itself. In testing the IMU may exhibit this issue after approximately an hour of continuous use. When the drift occurs, re-calibration may be required.

[0072] Accordingly, to resolve these two negative aspects of the IMU based motion capture system additional inputs to the system may be used both for correction and real time recalibration. These typically involve IR emitters at the camera and IR reflective light sources placed strategically on the subject. This requires both high output IR illumination and high speed camera systems, traditionally more than eight imagers to resolve high resolution marker points in a 3D volume.

[0073] Referring now to FIG. 9, an embodiment consistent with virtual reality haptic feedback process 10 is provided. This example differs from those above in several ways. First, there is no assumed fixed imaging device network setup whose position is determined a priori. Second, this approach doesn't assume that there will be any emitters on the imaging device(s) that will be assisting the calibration process. Finally, this approach uses a pulse encoded free space optical transmission to quickly identify position tracker uniquely to each point in space. This approach capitalizes on the pre existing deployment of a time of flight (ToF) tracking system that could be used in conjunction with the IMU based system, but may utilize an imaging sensor with sensitivity to the specific wavelength of emitted light from prepositioned LEDs on the tracking units themselves. When the points are in the field of view (FOV) of the imager they may be identified, their relative positions may be calculated either through multiple imagers whose relative 3D locations may be determined at the initial calibration time, or through the use of Time of Flight (ToF) imaging devices that may calibrate through either fixed location or mounting on the body. The key with this approach is that the system may be constantly readjusting opportunistically as the pulsed illuminators come in and out of view of any acceptable calibration imaging technology.

[0074] In some embodiments, the pulse encoding for transmission is a form of open space visible or non-visible (Infrared) communications channel. This combined with both point tracking from the imager as well as the relative 3D space inverse calculations from these observed points allow the calibration to occur in real time. The onboard MCU may pulse an encoded signal that may be observed by the imaging device to determine which sensor the point represents. This can be accomplished quickly using high speed pulsed LED transmission in either visible light spectrum or IR spectrum depending on the desired imaging capture device. The calibration sequence can be periodically fired, or controlled by the host computer to assist in initial calibration. Once the system has determined the identify of an illuminated point a high speed tracking algorithm determines the 3D position an evaluates this against the expected 3D location that is continuously updated as part of the IMU tracking. If a discrepancy between the expected and observed location is found, adjustments can be made to coax the calibration data to a ground truth state, or in extreme cases, the user can be notified that a brief recalibration process may be required.

[0075] In some embodiments, the Inter- Integrated Circuit (I2C) protocol may be employed to connect many physically separated sensors. An addressing problem surfaces when several identical sensors may be connected on the same I2C bus. Sensors that have I2C interfaces typically have only one or two addresses. For example, the Invensense MPU-9250 Motion Tracking Device includes approximately sixteen MPU- 9250 devices are needed to instrument a hand for virtual reality interfacing. However, the MPU-9250 only has two possible addresses. The MPU-9250 also has a Serial Peripheral Interface (SPI) capability. Using this interface, however, adds the complexity of requiring a separate chip select line for each sensor.

[0076] Accordingly, a solution for using many sensors on a single I2C network is to incorporate a microcontroller interface as shown in FIG. 10, which depicts a generic sensor connection to an I2C Bus. The sensor board microcontroller may dedicate an I2C module for the controller bus and have additional resources to communicate with the sensor device. For the MPU-9250 example, the sensor board microcontroller may communicate on the controller network using I2C and communicate with the MPU- 9250 sensor using SPI or I2C. In this arrangement, the sensor interface may be isolated from other network devices. This approach works well but becomes burdensome when connecting more than a few sensor boards. The scalability problem involves assigning a unique I2C address to each microcontroller. One method is to set the address by sensing General Purpose Input Output (GPIO) line settings at startup. Two pins dedicated for addressing may identify four microcontrollers. Three pins may identify eight microcontrollers and so on. This method takes additional microcontroller hardware resources and additional board space for biasing the pins. In another method, the address may be compiled directly into the executable code. Each microcontroller requires a unique executable file. Either method can easily work for small networks.

[0077] In some embodiments, dynamically assigning I2C addresses in larger networks can be achieved by using an additional GPIO line and defining a suitable protocol. In addition to solving the scalability problem this protocol may also be used to enable exception processing, fault recovery, device“hot-swapping”. Devices in accordance with the present disclosure may include a microcontroller including enough hardware resources to dedicate an Inter-Integrated Circuit (I2C) communications module and two General Purpose Input Output (GPIO) pins to implement the protocol. Embodiments included herein use a proprietary virtual reality communications protocol to assign a unique identifier to each device in the network. The architecture is shown in FIG. 11, which depicts an example network overview. The controller may be configured as an I2C master and the protocol may be implemented to assign a unique I2C address to each device. When network setup is completed the protocol may be used to manage network exceptions.

[0078] In some embodiments, the controller may use three signals to control the network. The Serial Data (SDA) and Serial Clock (SCL) are I2C lines that may be connected to each device and in a standard I2C network. The third signal may be a Neighbor Bus (NB) signal that connects to one of the two Neighbor Bus signals on Device 1. The other neighbor bus signal on Device 1 may be connected to one of the neighbor bus signals on Device 2. FIG. 11 shows Neighbor Bus 1 (NB_l) of one device connecting to NB_0 of the next device. This is not a constraint. Either neighbor bus of one device can connect to either neighbor bus of the next device. Devices may be connected in this manner to a maximum of 126 devices. The I2C 7-bit address space allows 128 addresses and addresses 0x00 and 0x7F may be reserved for the protocol. There may be additional addresses reserved by the I2C specification but those reservations are ignored by this protocol.

[0079] In some embodiments, the four states shown in FIG. 12, may be executed sequentially. When a network device is initialized it allocates five memory bytes to the protocol and enough memory for the controller and device to exchange information. The I2C module may not be active so it does not respond to I2C commands. Both of the neighbor bus pins may be configured as inputs so that the pull up resistors pull the neighbor bus signal HIGH. The controller may initiate the protocol by pulling its neighbor bus LOW. Device 1 detects this HIGH-to-LOW transition and assigns the pin on which the transition occurred as its upstream neighbor bus (US_NB). It assigns the other neighbor nus pin as its downstream neighbor bus (DS_NB). It sets its state to RESET and pulls its DS_NB LOW. This transition signals Device 2 to accomplish the same tasks and the action continues until all devices are in the RESET state. The controller may release its neighbor bus so that the signal floats HIGH. Device 1 detects this LOW-to-HIGH transition and initializes its I2C module as a slave device with address 0x7f. Device 1 sets its state to SETUP All other devices remain in RESET. The controller may read the contents of Register 1 on the I2C slave that has address 0x7F. Since only Device 1 is configured with this address it responds by transmitting the value of Register 1. When the controller verifies the value, it marks Device 1 as PRESENT and transmits a unique I2C address to it. Device 1 detects that it received an address from the controller and reconfigures it I2C module as a slave with unique address. It sets its state to ACTIVE and releases its DS_NB. Device 2 recognizes the LOW-to-HIGH transition on its UP_NB and enters SETUP. The Controller again reads Register 1 from the slave device with address 0x7F. Since Device 2 is the only device with this address, it responds by sending the value of Register 1. Upon VR, the controller assigns a unique I2C address to Device 2. Device 2 reconfigures its I2C module with the new address and releases its DS_NB so that it float HIGH. This sequence continues until no device responds to the Controller’ s query to address 0x7F. All devices are ACTIVE. Devices in the ACTIVE state update register addresses with sensor values and respond to the controllers read and write commands.

[0080] In some embodiments, the protocol may be implemented in a single master I2C network; therefore, the slave devices cannot initiate I2C communications. Several methods are implemented by which a device can signal an exception. Embodiments included herein may implement the basic exception protocol to request action from the controller. When a device encounters a condition that warrants action from the controller it sends a 1 millisecond pulse on its US_NB. The upstream neighbor detects this pulse and sends a pulse on its US_NB. The pulse may propagate upstream until the controller receives the signal. The only information conveyed in the pulse is that one of the network devices requested servicing. The controller queries each device in priority order to determine which device initiated the exception.

[0081] In some embodiments, the basic exception protocol may not detect problems with downstream devices. It only responds to requests from downstream devices. For example, the basic protocol does not detect disabled devices such as physically disconnected or malfunctioning devices. Accordingly, an advanced exception protocol may be implemented to include downstream monitoring. Adding or replacing devices on an active network may be possible if the network implements an advanced exception protocol. When a device that implements the advanced exception protocol enters the Active state, it monitors its DS_NB for a signal and also sends a signal on its US_NB. When a device does not receive the expected signal on its DS_NB it may initiate an exception signal on its US_NB that is propagated to the controller. Different versions of the advanced exception protocol implement neighbor bus signaling that is appropriate for the network.

[0082] In some embodiments, this signaling method uses a voltage divider on both the US_NB and the DS_NB of each device as shown in FIG. 15. The devices monitor the voltage on the DS_NB using an analog to digital converter or a comparator. When the expected voltage is not detected, a software exception signal is propagated to the Controller. This signaling can be used to accommodate“hot swapping” devices. When a device is removed from an active network, the controller may detect where the break occurs and discontinues polling devices beyond the break. When a device is reconnected, or added to the end of a network, the neighbor bus voltage level may be detected by the upstream neighbor that initiates an exception to the controller.

[0083] In some embodiments, several software signals can be implemented in the advanced exception protocol. The simplest is a periodic pulse that a device sends on its US_NB. The upstream neighbor detects the presence/absence of the signal and initiates an exception as required. A device watchdog timer can be implemented to monitor the signal’s presence. A pulse width modulated signal may also be used by the advanced exception protocol. As with a pulse, the device monitors the presence/absence of a PWM signal with a specified duty cycle. PWM signals with differing duty cycles can be used to propagate more information than a single pulse can. A unique advanced exception protocol implementation is for a Device to interrupt the normal I2C traffic and communicate directly with the Controller. The Device may interrupt the normal I2C communications by using a GPIO to pull the I2C SCL line LOW. No I2C communication occurs when the clock is held LOW. In this condition the I2C data line is idle and can be used for simplex asynchronous communication. FIGS. 16-19, show an asynchronous communication using the I2C Data Line, an electrical schematic for position tracker board, a physical schematic for position tracker board, and a position tracker board respectively.

[0084] Embodiments of virtual reality haptic feedback process 10 may include a real time haptics mapping engine and toolset for MR/VR training simulations. Embodiments may also include a point-of-need delivery service for simulated, high- fidelity, training environments through the use of MR/VR and haptic surrogates. Embodiments may also include a cross-platform, high-performance, high-fidelity, scalable presentation client capable of supporting real-time haptics mapping in simulated training environments. Embodiments may also include a positional tracking, synthetic proprioception and haptic feedback system.

[0085] In some embodiments, the teachings of the present disclosure may increase 3D simulation training availability and effectiveness while significantly reducing capital costs associated with 3D motion simulation training systems. Embodiments may also include significantly improved motion tracking for more direct representation of physical interactions and a significant improvement to current state of the art initialization, calibration and setup.

[0086] Initial development of haptic feedback involved the use of Eccentric Rotational Mass (ERM) vibration motors as haptics feedback drivers. These drivers are small vibratory devices that produce a strong vibration when an electrical signal is applied to them. While the main benefit of ERM drivers is their low cost and omni-directional feedback, they do have several negative aspects. The first negative aspect is related to the spool up time of the driver itself. Since the vibration may be produced by a physical effect of the rotation mass there is a time delay in the full magnitude felt force of the ERM. This time delay is potentially significant enough in both spin up and spool down time that the haptic feedback may not match the point of perceived“touch” with the VR object. The second negative aspect of the ERM haptic approach is in the variability of force. Variable vibratory force may be produced by varying the voltage input to a simple driver circuit through the use of a pulse width modulation input to a transistor that drives the ERM. This essentially reduces the voltage to the device which in turn reduces the speed of the motor and ultimately the felt or perceived force of the vibration itself. This approach has the negative impact of further delaying the target maximum spin up force of the unit and can introduce non-linearity in the voltage input to vibratory force magnitude response curve that requires additional calibration. Finally due to the physical requirements of the rotating mass, the power consumption (current draw) of these devices, while acceptable isn’t ideal for long term battery powered use.

[0087] In an effort to avoid these negative aspects of the ERM design approach, alternative devices may be used as drivers for the haptic response. Of these, it was found that piezo electric drivers could be used to impart a felt force and had many positive traits that held specific benefit for this application. The quick response time of the piezo drivers was found to be particularly advantageous. In addition, the piezo driver may be cycled quickly to simulate the vibration produced by the ERM circuit. Finally, commercial piezo drivers can be chosen that provide a deflection in a singular axis and oriented such that they“push” in a direction that mimics a tapping or touching feeling. If oriented around the finger properly, piezo drivers can simulate a touch haptic feedback along any point of the finger which in turn can simulate the flipping of switches or pressing of buttons in a non-traditional (not fingertip) manner. Accordingly, the piezo drivers may be used to simulate a force by providing a displacement whereas the ERMs mimic a force using a vibration. Piezos may also be used to cause a vibration by oscillating the displacement at high speed.

[0088] In some embodiments, the piezo electric driver response may be used to simulate the actual force/distance or force/time response curve of different objects. For example, the diagram shown in FIG. 20 depicts the force response over distance for a typical keyboard key press. This curve has a different response for the downward press as it is a release curve but both the downward stroke and rebound impart force on the tip of the finger. Once the capture of the response curve for multiple devices has been made, these curves can be stored and reused to mimic real world haptic interactions in the virtual reality space. In the case of the keyboard key press, if a user interfaces with this device in a virtual space, the keypress“Waveform” may be loaded and played back to the user with each keystroke. While the resistive force may not be possible to reproduce without exoskeletal rigging, the relative magnitude of the displacement, imparted by the piezo driver can be accurately reproduced for both the down stroke and release stroke of the key press. While these may be mapped to a force over travel distance for the stroke itself, the actual recorded measurement may be a voltage provided by the piezo sensor over a period of time. In order to measure the distance traveled, a known parameter of the object must be known such as the travel distance of the keyboard button in this case, or the velocity of the fingertip must be measured. Either method may be utilized to convert the voltage over time to a notional force over distance record.

[0089] In some embodiments, this approach may be applied to any number of devices such as switches, knobs, buttons, toggles, momentary buttons, triggers etc. The example shown in FIG. 21 is taken from a sealed tactile switch similar to those found in a waterproof industrial application. In this example the magnitude of the press is higher than that of the keyboard key and the total stroke distance is shorter (.35mm vs 6.5mm). This will result in a faster felt“click” with a higher magnitude when clicked in a virtual environment than that of a virtual keyboard.

[0090] In addition to the fixed playback of the force over distance playback the velocity vector of the finger motion itself can be used to vary the haptic response. In this case the fingertip velocity vector may be calculated from the positional tracking device and the resulting haptic response may be modified for this relative velocity of the fingertip. This in turn may be used to vary the response curve of a Force/time magnitude that can easily be used to drive to the piezo haptic circuit. Faster fingertip motion may result in keypresses that are replayed more quickly. The method of modification is quite straightforward with the replay speed being a factor of the current fingertip speed in relation to the original capture speed. [0091] Embodiments of virtual reality haptic feedback process 10 may be configured to use Electromyography (EMG) to bias the magnitude of haptic response. Efforts have been made to develop the ability to provide variable vibrational intensity during haptic interactions based on volitional control of the user with biofeedback. Embodiments included herein may be configured to utilize an electrical output measurement device (e.g., an armband) and the EMG feedback from this unit may be fed into the main game engine software. It should be noted that any other form of ECG/EKG type pad, or any system that can measure the electrical output caused by muscle contraction may be used without departing from the scope of the present disclosure. Some or all aspects of the measurement device may be directly interfaced in Unity3D with C# support. Implementation of biofeedback in the form of electromyography or EMG may allow the user to perform normal gestures and may scale the vibrational feedback based on the intensity of the movement. The electrical output measurement device armband provides direct access to eight channels of EMG to measure the magnitude of muscle activity. The muscle activation from specified gestures can be used to control the oscillation frequency of the eccentric rotating mass or for piezo haptic feedback circuits. This provides a mechanism to elicit feedback proportional to the bio-signals which can enhance the overall experience and fidelity of the feedback. The research efforts with the electrical output measurement device armband concern the ability to utilize bio signals, accelerometer, and positional data to scale the frequency of the vibrotactile sensation. The electrical output measurement device armband may be configured to incorporate Bluetooth technology and lithium ion batteries to maintain an untethered and wireless approach to collect data and facilitate integration with other virtual reality components. These positive features were utilized to develop comprehensive virtual environments that support upper extremity tracking and incorporate biofeedback interaction.

[0092] Referring now to FIGS. 22-25, embodiments of the present disclosure depicting user interfaces and displays consistent with the present disclosure are provided. Interactive virtual environments have been created to illustrate and test the capabilities of natural gestures and measure the associated muscle activity. FIG. 22 depicts an EMG reticle simulation with support for EMG threshold training to identify applicable forces and interactions during virtual interaction. FIG. 23 depicts an alpha numeric keypad interaction test with EMG data recorded from the electrical output measurement device. Proportional vibrational response generated from EMG activity with haptic device. FIG. 24 depicts a multi-functional display (MFD) interaction test with associated EMG data (left). FIG. 25 depicts a cyclic interaction test with associated EMG used to control vibrational response. The electrical output measurement device armband was incorporated to quantify muscle activity of the forearm to create a threshold range for each virtual activity and interaction. The implementation of EMG thresholds provides the ability to monitor user muscle activity and increase the fidelity of interaction and engagement. Implementation of virtual EMG thresholds complement the vibrational feedback to provide a proportional response to the muscle activity in the form of vibration. Complementary visual feedback may be provided in the form of explosions to notify the user that an excessive amount of force was utilized. The combinatory approach with haptic and visual feedback paired with the electrical output measurement device armband provide a complementary immersive system that can maintain and monitor force application. Assessments were conducted in the virtual cockpit to determine the overall muscle response during haptic interactions. Haptic interactions were measured with the electrical output measurement device to quantify EMG activity and assess digital interaction with three scenarios: alphanumeric keypad, multi functional display, and flight cyclic. Negative EMG forces were filtered and positive amplitudes were mapped to vibrational feedback to create the proportional response.

[0093] In some embodiments, measured gestures and interactions were utilized to create an EMG threshold simulation. The EMG simulation was designed to respond to user muscle activity. The electrical output measurement device armband was used to track the movement of the upper extremity based on positional data from the accelerometer and IMU. Positional data was utilized in Unity to control a reticle. Participants were required to locate and destroy the target by identifying the respective color-coded target with the appropriate reticle and performing a specific gesture (fist, wave, or tap) to generate desired EMG. Once desired EMG threshold was reached, the reticle released a missile at the target. EMG simulation was designed to support both left and right hand to train multiple hands or engage multiple users. The EMG simulation represents an application that utilizes real-time EMG to provide immersive simulations that can be correlated to muscle activity and implemented in realistic virtual environments like the AH-64. This work represents the initial implementation of biofeedback where digital force application and digital interactions can be utilized to increase the fidelity of the virtual environment with appropriate physics and physiological control.

[0094] FIG. 26 depicts an example showing EMG muscle activity during interaction with digital cyclic flight stick. EMG activity for each channel (eight) measured with electrical output measurement device during respective interaction for cylindrical grasp movements. Activity measured represents isometric movement for grasp movement. FIG. 27 shows an example showing haptic device paired with electrical output measurement device armband to provide biofeedback with proportional vibration based on EMG muscle activity.

[0095] Some portions of the preceding detailed description have been presented in terms of algorithms and symbolic representations of operations on data bits within a computer memory. These algorithmic descriptions and representations are the tools used by those skilled in the data processing arts to most effectively convey the substance of their work to others skilled in the art. An algorithm is here, and generally, conceived to be a self-consistent sequence of operations leading to a desired result.

[0096] The operations are those requiring physical manipulations of physical quantities. Usually, though not necessarily, these quantities take the form of electrical or magnetic signals capable of being stored, transferred, combined, compared, and otherwise manipulated. It has proven convenient at times, principally for reasons of common usage, to refer to these signals as bits, values, elements, symbols, characters, terms, numbers, or the like.

[0097] It should be kept in mind, however, that all of these and similar terms are to be associated with the appropriate physical quantities and are merely convenient labels applied to these quantities. Unless specifically stated otherwise as apparent from the above discussion, it is appreciated that throughout the description, discussions utilizing terms such as "processing" or "computing" or "calculating" or "determining" or "displaying" or the like, refer to the action and processes of a computer system, or similar electronic computing device, that manipulates and transforms data represented as physical (electronic) quantities within the computer system's registers and memories into other data similarly represented as physical quantities within the computer system memories or registers or other such information storage, transmission or display devices.

[0098] When implemented as an apparatus for performing the operations described herein, the apparatus may be specially constructed for the required purposes, or it may comprise a general-purpose computer selectively activated or reconfigured by a computer program stored in the computer. Such a computer program may be stored in a computer readable storage medium, any type of storage media or device suitable for storing electronic instructions, and each coupled to a computer system bus.

[0099] The processes presented herein are not inherently related to any particular computer or other apparatus. Various general-purpose systems may be used with programs in accordance with the teachings herein, or it may prove convenient to construct a more specialized apparatus to perform the operations described.

[00100] When implemented in software, the elements of the embodiments of the invention are essentially the program, code segments, or instructions to perform the tasks. The program, code segments, or instructions can be stored in a processor readable medium or storage device that can be read and executed by a processor or other type of computing machine. The processor readable medium may include any storage medium or storage device that can store information in a form readable by a processor or other type of computing machine. The program or code segments may be downloaded via computer networks such as the Internet, Intranet, etc. and stored in the processor readable medium or storage device.

[00101] The embodiments of the invention are thus described. While embodiments of the invention have been particularly described, they should not be construed as limited by such embodiments. The embodiments of the invention should be construed according to the claims that follow below.