TECHNICAL FIELD

[0001] The present disclosure is directed, in general, to the use of augmented reality technologies for training applications and more specifically, to a wearable augmented reality training system for remote and group training.

BACKGROUND

[0002] Augmented Reality (AR) is a technology that displays digital information in the field of view of a user, mostly using head-mounted devices such as Microsoft's HoloLens® or Meta Company's Meta 2®. AR tracks the environment of a user, and in this way, a user is enabled to place virtual objects and scenes at fixed 3D positions in any given real- world environment.

[0003] A skilled industrial workforce is essential for a successful and productive industrial operation. In order to ensure a skilled and educated workforce, training of new workers is necessary.

SUMMARY

[0004] In one construction, a method of training a first trainee includes positioning a first wearable device on the first trainee, positioning the first trainee in a first training space, positioning a second wearable device on the trainer, positioning the trainer in a second training space, and operating a computer to generate elements within a virtual space. The method also includes communicating the virtual space and the elements to each of the first wearable device and the second wearable device, integrating the virtual space and the elements, the first training space, and the second training space, and utilizing the elements to facilitate the completion of a training task by the trainee under the guidance of the trainer.

[0005] In another construction, a system that facilitates a trainer training a trainee to perform a task includes a first wearable device operable to project 3D holographic virtual elements that make up a virtual space, the first wearable device worn by the trainer and a second wearable device operable to project 3D holographic virtual elements that correspond to the virtual space, the second wearable device worn by the trainee. A physical object is positioned in a field of view of one of the trainer and the trainee, and a computer is operable to integrate the physical device into the virtual space by generating a representation of the physical device. The computer integrates the first wearable device and the second wearable device so that the first wearable device and the second wearable device display the virtual space, the 3D holographic virtual elements, and one of the physical device and the representation of the physical device from the perspective of the trainee and the trainer.

[0006] The foregoing has outlined rather broadly the technical features of the present disclosure so that those skilled in the art may better understand the detailed description that follows.

Additional features and advantages of the disclosure will be described hereinafter that form the subject of the claims. Those skilled in the art will appreciate that they may readily use the conception and the specific embodiments disclosed as a basis for modifying or designing other structures for carrying out the same purposes of the present disclosure. Those skilled in the art will also realize that such equivalent constructions do not depart from the spirit and scope of the disclosure in its broadest form.

[0007] Also, before undertaking the Detailed Description below, it should be understood that various definitions for certain words and phrases are provided throughout this specification, and those of ordinary skill in the art will understand that such definitions apply in many, if not most, instances to prior as well as future uses of such defined words and phrases. While some terms may include a wide variety of embodiments, the appended claims may expressly limit these terms to specific embodiments.

BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Fig. 1 is a schematic illustration of an interactive and collaborative augmented reality training system.

[0009] Fig. 2 is a trainer's view of a trainee using a virtual tool to perform a repair step on a virtual robot. [0010] Fig. 3 is the trainee's view of the virtual tools and the virtual robot of Fig. 2.

[0011] Fig. 4 is a trainer and a trainee view of the same virtual space and virtual tools from their respective perspectives.

[0012] Fig. 5 is a trainer's view of a trainee performing a task when the trainer is remotely located.

[0013] Before any embodiments of the invention are explained in detail, it is to be understood that the invention is not limited in its application to the details of construction and the arrangement of components set forth in the following description or illustrated in the following drawings. The invention is capable of other embodiments and of being practiced or of being carried out in various ways. Also, it is to be understood that the phraseology and terminology used herein is for the purpose of description and should not be regarded as limiting.

DETAILED DESCRIPTION

[0014] Various technologies that pertain to systems and methods will now be described with reference to the drawings, where like reference numerals represent like elements throughout. The drawings discussed below, and the various embodiments used to describe the principles of the present disclosure in this patent document are by way of illustration only and should not be construed in any way to limit the scope of the disclosure. Those skilled in the art will understand that the principles of the present disclosure may be implemented in any suitably arranged apparatus. It is to be understood that functionality that is described as being carried out by certain system elements may be performed by multiple elements. Similarly, for instance, an element may be configured to perform functionality that is described as being carried out by multiple elements. The numerous innovative teachings of the present application will be described with reference to exemplary non-limiting embodiments.

[0015] Also, it should be understood that the words or phrases used herein should be construed broadly, unless expressly limited in some examples. For example, the terms "include" and "comprise," as well as derivatives thereof, mean inclusion without limitation. The singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. Further, the term "and/or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items. The term "or" is inclusive, meaning and/or, unless the context clearly indicates otherwise. The phrases

"associated with" and "associated therewith," as well as derivatives thereof, may mean to include, be included within, interconnect with, contain, be contained within, connect to or with, couple to or with, be communicable with, cooperate with, interleave, juxtapose, be proximate to, be bound to or with, have, have a property of, or the like.

[0016] Also, although the terms "first", "second", "third" and so forth may be used herein to refer to various elements, information, functions, or acts, these elements, information, functions, or acts should not be limited by these terms. Rather these numeral adjectives are used to distinguish different elements, information, functions or acts from each other. For example, a first element, information, function, or act could be termed a second element, information, function, or act, and, similarly, a second element, information, function, or act could be termed a first element, information, function, or act, without departing from the scope of the present disclosure.

[0017] In addition, the term "adjacent to" may mean: that an element is relatively near to but not in contact with a further element; or that the element is in contact with the further portion, unless the context clearly indicates otherwise. Further, the phrase "based on" is intended to mean "based, at least in part, on" unless explicitly stated otherwise.

[0018] Factory jobs often require a skilled and trained workforce. Often, even skilled workers do not have the necessary qualifications or skills and need to receive extensive training to work on specialized equipment or tasks. Skilled veteran workers are a great source of knowledge and can mentor young workers in training. However, hands-on training may require special or expensive hardware that might not always be available and/or is too expensive to provide in the necessary quantities for training. In addition, hands-on training is sometimes forgotten as it is difficult for trainees to take notes while learning practical hands-on procedures.

[0019] With reference to Fig. 1, an AR (Augmented Reality) system 10 (sometimes referred to as Mixed Reality or MR) can be used to enhance collaborative and interactive training between a trainer 15 and one or more trainees 20. The system 10 and method described herein allows trainers 15 to train trainees 20 with 3D virtual representations of real- world objects as well as with real world objects. Fig. 1 schematically illustrates a trainer 15 and a trainee 20 positioned in the same physical training space 25 (shown in Fig, 2) and wearing head-mounted devices 30, 35 such as Microsoft's HoloLens® or Meta Company's Meta 2®. The head mounted devices 30, 35 operate to project 3D holographic elements or objects (objects and elements are used interchangeably) that define a virtual training space into a user's field of view. The head- mounted devices 30, 35 also allow the wearer or user to see the actual physical environment around them such that the 3D objects or elements are projected onto or into the actual physical environment. The head-mounted devices 30, 35 can also project 2D objects to one or both users.

[0020] Training is generally centered on fixing or building (assembly, disassembly, operation, maintenance) an object 40. In the construction illustrated in Fig. 2, the object 40 is a robot or robot arm. Of course, many other objects 40 could be the center of the training session. The object 40 is either a physical object 40a (i.e., it is a real object positioned in the room 25) or it is a virtual object 40b that is generated by the head-mounted devices 30, 35. In some

constructions, a computer 45 that is separate from the head-mounted devices 30, 35 generates one or more virtual objects 40b and/or facilitates communication between head- mounted devices 30, 35. It should be noted that typical head-mounted devices 30, 35 include a computer that is operable to run programs or apps that facilitate their operation. Generally, the computer 45, if utilized, acts as a server between the head-mounted devices 30, 35 and/or facilitates

communication. Hereinafter, the term computer includes any computer or combinations of computers that operate to perform a desired task or function. Often, the computer is part of one of the head-mounted devices 30, 35 or is made up of computers in both head-mounted devices 30, 35 working together. Additionally, computers external to the head-mounted devices 30, 35 can work alone, or with computers in the head-mounted devices 30, 35 to perform the desired tasks or functions.

[0021] In the case where the object 40 is a physical object 40a, the head-mounted display 30, 35 integrates the object 40a into the virtual space. The integration links the virtual space to the actual environment with the physical object 40a defining a perspective orientation with respect to both environments. In other words, the object 40a defines the perspective of each user 15, 20 and allows the head mounted displays 30, 35 to better display both actual objects 40a and virtual objects 40b more accurately. In the case where the object 40 is a virtual object 40b, the head mounted displays 30, 35 build and position the object 40b in the virtual environment. In some situations, only one of the trainer 15 or the trainee 20 is in the room 25 with the physical object 40a with the other being remotely located. In this situation, the physical object 40a is rendered as a virtual object 40b for the remote user.

[0022] Virtual elements 50 (sometimes referred to as virtual tools or tools) such as tools are generated by the head mounted displays 30, 35 for use in the training. Virtual tools 50 could include virtual screwdrivers, pliers, wrenches, hammers, etc. Of course physical tools could also be used with a physical object 40a or a virtual object 40b if desired.

[0023] In the training session, the trainer 15 establishes a task to be completed. In the illustrated construction, the task is displayed in the trainee's field of view as a 2D note 55. It is preferable that the 2D note 55 be pinned to a fixed position in the trainee's field of view so that it moves with the trainee 20. However, the task could be pinned to the object 40 or could be hidden by the trainee 20. The trainer 15 can provide one or more tools as virtual elements 50 for the trainee 20 to use. In the illustrated construction, four virtual tools 50 are provided. The training can be tailored to the level of the trainee 20. For example, the task could be broad (e.g., fix the robot) or could be a more narrow step by step process that guides the trainee 20. Similarly, the virtual tools 50 could be provided as a full tool kit including virtual tools 50 that are needed as well as virtual tools 50 that are not, or could be provided individually for each step to aid the trainee 20.

[0024] During the training session, the trainee 20 can ask the trainer 15 questions through the head-mounted devices 30, 35 and the trainer 15 can answer the questions through the head- mounted devices 30, 35. In addition to a simple text answer, the trainer 15 can place virtual notes 55 or indicators 60 on or near the object 40 to further guide the trainee 20. The use of text questioning within the virtual space is advantageous as it is possible to record the training session on a recording device 65, thereby capturing the text questions as well. Also the trainee can see the trainer's body language.

[0025] Fig. 2 illustrates the trainer's view 70 of a trainee 20 performing a task with both trainer 15 and trainee 20 in the same training space 25 or room. In Fig. 2, the object 40 is a physical object 40a but a virtual object 40b is also provided. While this would not be typical, it does illustrate both ways of displaying the object 40. The virtual tools 50 are visible and appear to be floating in the environment. The trainee 20 is grasping a pair of pliers while the trainer 15 has placed indicators 60, in the form of arrows, on the object 40a, 40b to show the trainee 20 where to use the tool 50. The trainee's view 75 of the arrangement of Fig. 2 is shown in Fig. 3 with the trainee 20 focused on the virtual object 40b rather than the physical object 40a. In Fig. 3, the available tools 50 are visible and again appear to be floating in front of the object 40b. In addition, the indicator 60 placed by the trainer 15 is clearly visible as a large arrow pointing at the area where the trainer 15 wants the trainee 20 to focus.

[0026] Fig. 4 illustrates the view through the head-mounted devices 30, 35 when the trainer 15 and the trainee 20 are in training spaces 25 that are separated from one another and there is no physical object 40a. In this arrangement the head mounted display 30, 35 generate a virtual object 45b (the robot arm) and a virtual tool 50 in the form a screwdriver. Both the trainer 15 and the trainee 20 see the virtual objects 50 but the perspectives are slightly different, as if the trainer 15 is watching over the shoulder of the trainee 20. Of course, the identical perspective could be provided to both the trainer 15 and the trainee 20 as well. Again, the trainer 15 has placed an indicator 60 near the screwdriver to aid the trainee 20. Because the trainer 15 and the trainee 20 are in different locations, the background (i.e., the physical environment) will be different. If the physical object 40a had been present in one of the training spaces 25, the trainer 15 or trainee 20 in that physical space 25 would see the physical object 40a while the other of the trainer 15 and the trainee 20 would see the virtual object 40b constructed by the computer 45 to match the physical object 40a.

[0027] In preferred constructions, when the object 40 and the elements 50 or tools are virtual the trainer 15 or trainee 20 can easily change his or her perspective. Fig. 5 illustrates the situation of Fig. 4 after the trainer 15 has changed perspective. Again, the screwdriver is present as a virtual tool 50 that appears to be floating near the virtual object 40b. The trainer 15 has placed a different indicator 60 on the environment to aid the trainee 20. The trainee 20 is represented as a virtual person 80 reaching for the virtual tool 50. The virtual image 80 of the trainee 20 is extrapolated from the image of the trainee's hand in the trainee's head-mounted device 35. In addition, the trainee 20 would have a virtual view of the trainer 15 in some constructions. [0028] While Figs. 1-5 have been described as being one-on-one training with a single trainer 15 and a single trainee 20, the system 10 is flexible enough to allow multiple trainers 15 and/or multiple trainees 20 to simultaneously participate in a training session. The most common situation would be a single trainer 15 and multiple trainees 20. In this situation, the trainer 15 would have the ability to communicate with each of the trainees 20 via the head-mounted devices 30, 35 either simultaneously as a group or individually. Thus, the trainer 15 could communicate the task to the entire class of trainees 20 and then provide notes 55, virtual elements 50, or indicators 60 to individual trainees 20 as required to assist them, or to the entire class of trainees 20.

[0029] In addition, the recording device 65 allows each training session to be recorded from the perspective of any party involved. Thus, trainees 20 could record and review their sessions at their leisure. As they don't need to hold an extra device in their hands.

[0030] In summary, trainers 15 and trainees 20 can see the same virtual representation of the object 40b as well as their real-world partner (trainer 15 / trainee 20). Feedback, given verbally or through gestures by the trainer 15 during a task, is received by the trainee 20 in real-time. The ability to interact naturally through gestures and voice with a virtual object 40b and the trainer 15 or trainee 20 makes the training very immersive and intuitive. Since the training typically uses a virtual object 40b and/or virtual tools 50, real-world objects 40a cannot be damaged and tasks can be repeated as often as necessary. Also, if a real object is not available or is expensive, training can still be performed. With the holographic feature of wearable AR devices or head- mounted devices 30, 35, a trainee 20 can also switch their vision to the point of view of the trainer 15. Training sessions can be recorded as a sight-tracking video through the head- mounted device 30, 35 and the trainee 20 can replay the recording multiple times and learn from the corrections made by the trainer 15. The trainer 15 can also leave virtual notes 55 that can help and give correction or support at crucial steps. The notes 55 will be stored in the individual session for the trainee 20 and the task can be revisited by the trainee 20 before repeating the same task. Also, two or more trainees 20 can learn collaboratively and work on exercises together. Scenarios include but are not limited to: maintenance, repair, replacement, replenishment, and exchange of parts. [0031] In one example of a training session, a trainer 15, wearing an AR device such as the head-mounted device 30, chooses a lecture or training scenario with a gesture or voice command from a collection of scenarios, which might include the repair of a robotic arm, for example. The trainer 15 places the robotic arm with voice command and/or gestures in the training room 25 as an object 40b and starts a training session for the trainee 20.

[0032] Either the trainer 15 describes the task of the selected scenario verbally to the trainee 20 or the trainee 20 has access to a written task description in a note 55 that is visible in the head- mounted device 35 of the trainee 20. Both trainer 15 and trainee 20 share the same view of the virtual object 40b and can walk around the object 40b, which allows them to see the object 40b or problem from various angles. While the trainee 20 is working on the task, the trainer 15 can see in his or her head- mounted device 30 exactly what the trainee 20 is doing and how he/she is approaching the task, as well as which tools 50 the trainee 20 has selected. Different levels of support or guidance can be offered to the trainee 20, depending on his or her progress with the task or his or her level of experience or training.

[0033] Tools 50 can be either selected from a menu or inserted by voice command. The trainer 15 can give feedback at any time during the task. If the trainee 20 is stuck, the trainer 15 can directly show the trainee 20 what to do or which tool 50 to use. The trainee 20 can experience the performance of the task through the eyes of the trainer 15 by directly streaming the head- mounted device 30 of the trainer 15 to the trainee 20 while the trainer 15 performs the task. Both trainer 15 and trainee 20 can see the virtual representation of the robotic arm 40b, the inserted tools 50, and the real person including any actions and movements, such as pointing or rotation. The trainee 20 can record the session with the head-mounted device 35 and watch the recording as part of the learning process.

[0034] Thus, the system 10 described herein enables two or more users to conduct collaborative and interactive training sessions for industrial use cases such as repair, replacement or maintenance. Augmented Reality is utilized to display virtual objects 40b, machines 40a or industrial factory floors as well as tools 50 and instructions for virtual training sessions. A recoding/replay feature and a feature that enables a trainee 20 to see the trainer's view support immersive training. [0035] As used herein, the terms "element" or "object" are sometimes used interchangeably and should be interpreted to refer to any virtual object generated by the computer 45 or the head- mounted devices 30, 35. Elements or objects can include 2D, 3D, and holographic objects or elements that are placed in the training space or in view of the head- mounted devices 30, 35.

[0036] Although an exemplary embodiment of the present disclosure has been described in detail, those skilled in the art will understand that various changes, substitutions, variations, and improvements disclosed herein may be made without departing from the spirit and scope of the disclosure in its broadest form.

[0037] None of the description in the present application should be read as implying that any particular element, step, act, or function is an essential element, which must be included in the claim scope: the scope of patented subject matter is defined only by the allowed claims.

Moreover, none of these claims are intended to invoke a means plus function claim construction unless the exact words "means for" are followed by a participle.