Field of Invention

This invention relates to the field of machine operations and controls and more specifically to interactive, real time displays for the operation of machines and machine tools and handling of related work pieces.

Background of the Invention

Modern machine tools are complex devices used to shape a wide variety of raw materials into finished products. The actions of these machines are controlled by a computer (Computer Numerical Control or CNC) to ensure precision and accuracy, but also require an operator to handle the material and ensure that the machine is working properly.

This computer, and its corresponding display, is contained in a CNC panel attached to the machine. This control panel displays valuable information about the process being done, and also allows the operator to change any number of inputs, such as the program for what part is being made or which tool should be used. However, this panel is generally not located in the same place that the operator must do the majority of his work. This creates a source of inefficiency, as the operator must walk to and from the panel during the work process, and means that the operator must look away from the part (and their hands) to see information on the panel. This creates a potentially dangerous situation, and is clearly not ideal.

The inefficient location of the CNC panel is one of the main reasons why production using machines has always required highly skilled workers to operate the machines in order to produce accurate results. These workers must be trained not only in how to operate each specific machine but also in how to create each specific part. The worker must either memorize the instructions for each part or continually return to the CNC panel to read the instructions which is highly inefficient. This is both expensive and time consuming, and requires high quality employees.

What is needed is a system that will significantly reduce the amount of training required for a worker. The device should clearly display the instructions required to make a specific piece directly in front of the worker. In this way the need for workers to learn how to produce each part can be eliminated. Further, the device should display this instructional information in the same location that the operator needs to be located in to perform the task. This instructional information will effectively guide the machine operator around the machine and provide instructions as to exactly what to do at each point. In this way, operators with less training and that are less skilled will be able to do the job of today's specifically trained workers, eliminating the need for training the worker for each specific part. By displaying exact instructions at a machine operating station, at the time they are needed, production errors can be minimized. The combination of these benefits permits a substantial reduction in required training and skill levels while still providing consistent, accurate results.

One possible solution is Augmented Reality (AR). Augmented Reality is a technology that allows us to enhance the world we live in by adding images and information to the world we see around us and allowing users to interact with these images. AR is a great tool for manufacturing using machine tools. Being able superimpose information (such as tooling or material handling instructions) directly onto the work area eliminates the need for the worker to step away from the work area to see information on the CNC panel and make changes to the program being executed. Current AR devices use a camera to get an image of the world and then

superimpose an image on top of that. Current common implementations include headsets and tablet devices. In either case the camera must continuously read images of the world as the user moves the camera around their surroundings and correctly create a 3D map of the objects in view. The software must also determine what these objects are in order to display information that is relevant to that object. However, current AR implementations are not ideal for a manufacturing work environment.

Tablet devices have the drawback of requiring the user to hold the camera and aim it at the work piece. This means that the worker can either use the AR device to get information or handle the material during the process, but not do both at the same time. Headsets do not have this limitation and allow the worker to work and see the real-time information freely, but share some important drawbacks with tablet devices. Both devices have limited power supplies that must be recharged, and could be dropped, broken, or misplaced. In a machine tooling environment relying on a fragile device like a tablet or headset is not ideal.

We need a device that can provide all of the benefits of AR without the drawbacks of current implementations. This device should allow for the display of images onto objects a user wishes to interact with, just like an AR system. The device should further allow for the user to interact with these images through gesture recognition. In a manufacturing environment the use of gesture recognition allows the user to interact with the displayed images without needing to step away from a work area or set down a work piece. The device should additionally include a safety interlock device to prevent the gesture control device from incorrectly reading the movements of a user. This safety device would require the user to make an additional step before any critical action is taken. If a gesture is misread by a machine tool this could result in a damaged work piece or even bodily harm to the user, so this is a vitally important addition to any gesture control system.

Various inventions have been developed to address these issues. U.S. Patent Application No. 2014/0293039, published for Appleyard et ah, discloses an imaging and safety and method for industrial machine. This reference applies to press brakes and relates to a light beam safety system for automatically determining the status of material being worked by the industrial machine. The combined imaging and safety system includes a light transmitter, one with a corresponding receiver and is intended to determine information regarding the tool and/or tool position and/or extent of work being carried out on a work piece. A shadow screen may be used to form an image of a profile of the tool tip on the end of the work place for assist in determining important information about the progress of the work being accomplished as it relates to a safety application for halting a machine or tool when a light beam is interrupted.

U.S. Patent Application No. 2014/0272775, published for Monty, is directed to a system and method for imaging in laser dental treatment. The device described in this reference includes a computation systems to compute a contour (2D) of a surface of the dental treatment area based on the image obtained by the imaging system and transforming a 3D scan to a 2D image on a computerized numerical control (CNC) milling machine that is used for generating a suitable dental restoration component such as a dental crown.

U.S. Patent Application No. 2014/0081444, published for Rudberg et ah, discloses a sequencer system for an automatic tape laying and projection system. Specifically, the system is utilized for building a part using automatic fiber placement and laser projection verification utilizing a CNC control machine for placing carbon fiber tape to a surface member to produce a layered composite part. The system comprises a fiber placement machine operating under CNC control that is use to place fibers relative to a surface in a fiber-laying process.

U.S. Patent Application No. 2014/0230026, published for Forero, is directed to a biometric -based access control system comprising a field communication link. This publication relates to the "motion capture system" component of the system in which various gestures and movement by the user are utilized to control an industrial machine under control of a CNC system. The invention is responsive to one or more touch gestures or a touch sensitive device such as a touch screen which detects user biometric information unique to that user and utilizes set information in carrying out a particular operation in the controlled machine. The user biometric information may comprise dynamic information such as an array of coordinate values corresponding to the touch gesture as well as the pressure values corresponding to the pressure used by the operator and includes such information as keystroke dynamics, motion dynamics and patterns. The system described may be utilized in both touch screen and touch pad devices.

U.S. Patent Application No. 2012/0290121, published for Gronbach, discloses a device for operating an automated machine for handling, assembling or machining work pieces. Specifically, this reference is intended to describe operating apparatus that is capable of inputting manual operator actions and for inputting contact-free operator actions. The apparatus for contact-free operator actions includes a sensor system for detecting contact-free actions of the operator and for picking up command-triggering operating actions of the operator by means of a contact-free operator actions detected picking up operating action of the operator utilizing contact-free operator actions detected by the sensor. U.S. Patent No. 8,638,984, issued to Roithmeier, is directed to a display of results of a measurement of work pieces as a function of the detection of the gesture of a user. This Patent addresses "motion capture system" and provides a gesture detection device which translates the movement of the user's hands to control the action of a machine on a work piece by means of Computer Numerical Control.

U.S. Patent No. 5,425,237, issued to Suer, discloses a counter balance system and is an example of an industrial machine that utilizes a pressure sensitive element to detect the hydraulic pressure in the system and thus compensate or control the operation of the system depending upon the limits set for the lubricant pressure.

It is an objective of the present invention to provide a system for use with machine tools, welders, press breaks and other industrial and medical equipment that allows the operator to remain nearby to his work station while receiving necessary inputs regarding his present operational tasks. It is another objective of the invention to coordinate these instructions with the CNC program of the machine. It is a further objective to have the system provide markings and limit lines directly on the work piece. It is still a further objective to allow the operator to control the machine with a series of gestures without requiring him to actually touch the machine controls. It is yet a further objective of the invention to provide at least one safety interlock to the operating features of the machine so that the machine operator cannot inadvertently cause the machine to function when not desired.

While some of the objectives of the present invention are disclosed in the prior art, none of the inventions found include all of the requirements identified. Summary of the Invention

The present invention addresses all of the deficiencies of prior art machine display and related method inventions and satisfies all of the objectives described above.

(1) A machine display providing all of the desired features can be fabricated from the following components. A machine is provided. The machine has at least one projectable surface. The projectable surface is located adjacent an operator workstation and adapted to reflect a projected image. A computer numerical control (CNC) computer is provided. The computer includes a CNC program. The program is adapted to control the machine. A CNC data file is provided. The data file is adapted to control performance of at least one operation by the machine. The data file includes positioning information for the machine. At least one calibrated projector is provided. The projector is calibrated to dimensions of and integrated with the machine, the CNC program and the data file. The projector projects instructional information for a machine operator at a projectable location on the projectable surface as the instructional information becomes relevant to a next machine operation. The instructional information includes any of text and properly sized and positioned images. The CNC program and the data file change the instructional information as operations are completed.

(2) In a variant of the invention, a work piece is provided. The work piece is sized, shaped and located to be used with the machine. The CNC data file is adapted to control performance of at least one operation by the machine upon the work piece. The at least one calibrated projector projects guide information for operations of the machine directly onto any of the projectable surface, the work piece and related tooling at a time the guide information is needed for a next operation. The CNC program and the data file change the guide information as operations are completed upon the work piece. (3) In another variant, the instructional information further includes an animation. The animation illustrates orientation of the work piece to the machine and movement of the work piece from a first work area to at least one subsequent work area. The animation includes any of properly sized and positioned images of said work piece and related tooling and real time sequencing of operational steps displayed upon any of a projectable area adjacent the work area and the work piece and the tooling. The animation is linked to the CNC data file and controlled by the CNC program. The animation serves as a work flow guide for the machine operator.

(4) In still another variant, the projector projecting the guide information is a laser projector. The laser projector is adapted to project a properly scaled image onto any three dimensional surface.

(5) In yet another variant, the guide information provides direct verification of dimensions of the work piece.

(6) In a further variant, a signal generating system is provided. The signal generating system generates signals linked to motions by the machine operator. A signal receiving system is provided. The signal receiving system receives signals from the signal generating system and the signals are converted into gestures which serve as inputs to control operations of the machine. Motions by the machine operator result in operations by the machine without physical contact between the operator and any machine controls. (7) In still a further variant, a signal generating system is provided. The signal generating system generates signals linked to motions by the machine operator. A signal receiving system is provided. The signal receiving system receives signals from the signal generating system and the signals are converted into gestures which serve as inputs to control operations of the machine upon the work piece. Motions by the machine operator result in operations by the machine without physical contact between the operator and any machine controls.

(8) In yet a further variant, the signal generating system further includes a pedestal. The pedestal includes a hand sensing camera. The hand sensing camera records hand motions of the machine operator and generates signals which are transmitted to the signal receiving system.

(9) In another variant of the invention, the signal generating system further includes a head mounted hand sensing camera. The hand sensing camera records hand motions of the machine operator and generates signals which are transmitted to the signal receiving system.

(10) In still another variant, the signal generating system includes a portable transmitter. The transmitter is sized and shaped to be manually manipulated by the machine operator and provides signals indicating the position of the transmitter relative to the machine. The signal receiving system includes a receiver. The receiver receives the signals from the transmitter and communicates the signals to a motion capture program. The motion capture program translates the signals into gestures which serve as inputs which are communicated to the CNC program to control operations of the machine.

(11) In yet another variant, the signal generating system includes a portable transmitter. The transmitter is sized and shaped to be manually manipulated by the machine operator and provide signals indicating the position of the transmitter relative to the work piece and the machine. The signal receiving system includes a receiver. The receiver receives the signals from the transmitter and communicates the signals to a motion capture program. The motion capture program translates the signals into gestures which serve as inputs which are communicated to the CNC program to control operations of the machine upon the work piece.

(12) In a further variant, the signal generating system is adapted for the machine operator to create annotations related to operation of the machine. The annotations are received by the signal receiving system and communicated to the CNC program. The CNC program saves the annotations to the CNC data file for future reference.

(13) In still a further variant, the signal generating system is adapted for the machine operator to create annotations related to either of the work piece or operations upon the work piece. The annotations are received by the signal receiving system and

communicated to the CNC program. The CNC program saves the annotations to the CNC data file for future reference.

(14) In still a further variant, the signal generating system includes a depth sensing camera. The camera generates signals describing the motions of the machine operator.

(15) In yet a further variant, the signal generating system includes a stereoscopic camera to generate signals describing the motions of the machine operator.

(16) In another variant of the invention, the signal generating system includes a remote device that has an embedded motion sensor, the remote device generating three dimensional signals describing the motions of the machine operator.

(17) In another variant of the invention, the embedded motion sensor is selected from the group that includes gyroscopes, accelerometers and compasses.

(18) In still another variant, the signal generating system includes a light curtain projected on the projectable surface and a camera system detecting machine operator motions impinging on the light curtain, the camera system transmits signals describing the motions of the machine operator. (19) In yet another variant, the signal generating system includes an infrared pen that contacts the projectable surface either on or adjacent to the machine and a camera system that detects machine operator motions indicated by movement of the infrared pen on the projectable surface. The camera system transmits signals describing the motions of the machine operator.

(20) In a further variant, the signal generating system includes a track pad. The track pad is located either on or adjacent to the machine and detects machine operator hand motions and transmits signals describing the hand motions of the machine operator.

(21) In still a further variant, a safety interlock device is provided. The interlock device requires an operator to perform a positive activity in order for the signals generated by the signal generating system to be implemented. This positive activity prevents accidentally initiated signals from being implemented.

(22) In yet a further variant, the safety interlock device includes a light curtain. The light curtain must be engaged by an operator for the machine to be activated, thereby providing a safety interlock.

(23) In another variant of the invention, the safety interlock device includes an infrared pen. A switch on the infrared pen must be engaged by an operator for the machine to be activated, thereby providing a safety interlock.

(24) In still another variant, the safety interlock device includes a pressure sensor. The pressure sensor provides a signal which controls activation of the machine.

(25) In yet another variant, the safety interlock device includes a strain gauge. The strain gauge provides a signal which controls activation of the machine.

(26) In a further variant, the safety interlock device includes a control switch. The switch is manually operable by the operator. A method of calibrating a machine display for guide information to a machine, includes the steps of:

providing a machine, the machine has at least one projectable surface, the projectable surface is located adjacent an operator workstation and adapted to reflecting a projected image;

providing a computer numerical control (CNC) computer, the computer includes a CNC program, the program adapted to control the machine;

providing a CNC data file, the data file adapted to control performance of at least one operation by the machine;

providing a laser projector, the laser projector is fixedly attached either to or adjacent to the machine;

attaching at least three reflective targets to the machine at locations where the laser projector can reach the targets;

providing a three dimensional mathematical model of the machine, the model includes three dimensional coordinates for each of the targets;

projecting a laser beam at each of the targets so that the beam reflects

directly back to the laser projector;

calibrating a position of the laser projector by comparing the three

dimensional coordinates of the targets from the three dimensional mathematical model with the positions of each of the targets detected by the laser beam; wherein the calibration permits the laser projector to project guide information onto the work piece and tooling for the work piece and an image of a work piece on the projectable surface at a known location in a designated size.

(28) A method of calibrating a machine display for instructional information to a machine, includes the steps of:

providing at least one optical projector, each of the optical projectors is fixedly attached either at or adjacent to a machine at an identified location point wherein each of the optical projectors can project upon a projectable surface of the machine;

identifying a fixed reference point on the projectable surface; projecting an alignment mark on the projectable surface;

aligning the projected alignment mark with the fixed reference point; entering coordinates of the alignment mark into a computerized screen alignment program;

projecting a rectangle upon the projectable surface;

identifying rectangular coordinates of each corner of the rectangle;

entering the rectangular coordinates into the screen alignment program; using the alignment program to determine skew correction factors for each projector wherein the optical projector can project a properly oriented image upon the projectable surface despite a non- orthogonal orientation to the projectable surface;

projecting horizontal and vertical lines upon the projectable surface, measuring the horizontal and vertical lines; entering the measurements into the screen alignment program;

computing horizontal and vertical scaling factors for each of the optical projectors;

wherein the optical projectors will employ the screen alignment program, the alignment mark, the skew correction factors, and scaling factors to project properly sized and positioned images upon the projectable surface.

(29) In a variant of the method of calibrating a machine display for instructional

information to a machine, further includes the steps of:

providing horizontal and vertical ruler projections upon the projectable surface; and

providing projected measurement points on the rulers.

(30) In a further variant of the invention, a machine control system includes a machine. The machine has a computer numerical control (CNC) computer. The computer includes a CNC program. The program is adapted to control the machine. A CNC data file is provided. The data file is adapted to control performance of at least one operation by the machine. The data file includes positioning information for the machine. The CNC program and the data file change as operations are completed. A signal generating system is provided. The signal generating system generates signals linked to motions by a machine operator. A signal receiving system is provided. The signal receiving system receives signals from the signal generating system. The signals are converted to gestures which serve as inputs to control operations of the machine. Motions by the machine operator result in operations by the machine without physical contact between the operator and any machine controls. (31) In still a further variant, a work piece is provided. The work piece is sized, shaped and located to be used with the machine. The CNC data file is adapted to control performance of at least one operation by the machine upon the work piece. The CNC program and the data file change as operations are completed upon the work piece.

(32) In yet a further variant, the signal generating system is adapted for the machine operator to create annotations related to operation of the machine. The annotations are received by the signal receiving system and communicated to the CNC program. The CNC program saves the annotations to the CNC data file for future reference.

(33) In another variant of the invention, the signal generating system includes a depth sensing camera. The camera generates data describing the motions of the machine operator.

(34) In still another variant, the signal generating system includes a stereoscopic camera. The camera generates data describing the motions of the machine operator. (35) In yet another variant, the signal generating system includes a remote device. The remote device has an embedded motion sensor. The remote device generates three dimensional data describing the motions of the machine operator.

(36) In a further variant, the embedded motion sensor is selected from the group that includes gyroscopes, accelerometers and compasses.

(37) In still a further variant, the signal generating system includes a light curtain projected on said projectable surface and a camera system that detects machine operator motions impinging on the light curtain. The camera system transmits signals describing the motions of the machine operator. (38) In yet a further variant, the signal generating system includes an infrared pen contacting said projectable surface either on or adjacent to the machine and a camera system that detects machine operator motions indicated by movement of the infrared pen on the projectable surface. The camera system transmits data describing the motions of the machine operator.

(39) In another variant of the invention, the signal generating system further includes a pedestal. The pedestal includes a hand sensing camera. The hand sensing camera records hand motions of the machine operator and generates signals which are transmitted to the signal receiving system.

(40) In still another variant, the signal generating system further includes a head mounted hand sensing camera. The hand sensing camera records hand motions of the machine operator and generates signals which are transmitted to the signal receiving system.

(41) In yet another variant, the signal generating system includes a track pad. The track pad is located either on or adjacent to the machine and detects machine operator hand motions and transmits data describing the motions of the machine operator.

(42) In a further variant, the machine further includes a safety interlock device. The interlock device requires an operator to perform a positive activity in order for the signals generated by the motion capture program to be implemented, thereby preventing accidentally initiated signals from being implemented.

(43) In still a further variant, the safety interlock device includes a light curtain. The light curtain must be engaged by an operator for the machine to be activated, thereby providing a safety interlock. (44) In yet a further variant, the safety interlock device includes an infrared pen. A switch on the infrared pen must be engaged by an operator for the machine to be activated, thereby providing a safety interlock.

(45) In another variant of the invention, the safety interlock device includes a pressure sensor. The pressure sensor provides a signal which controls activation of the machine.

(46) In still another variant, the safety interlock device includes a strain gauge. The strain gauge provides a signal which controls activation of the machine.

(47) In yet another variant, the safety interlock device includes a control switch. The switch is manually operable by the operator.

(48) A method of displaying instructional information for a machine includes the steps of:

providing a machine, the machine has at least one projectable surface, the projectable surface is located adjacent an operator workstation and adapted to reflecting a projected image;

providing a work piece, the work piece is sized, shaped and located to be used with the machine;

providing a computer numerical control (CNC) computer, the computer includes a CNC program, the program adapted to control the machine;

providing a CNC data file, the data file adapted to control performance of at least one operation by the machine upon the work piece, the data file includes positioning information for the machine and the work piece; providing at least one calibrated projector, the projector is calibrated to dimensions of and integrated with the machine, the CNC program and the data file, the projector adapted to projecting instructional information for an operator at a projectable location on the projectable surface as the instructional information becomes relevant to a next machine operation, the instructional information including any of textual data and properly sized and positioned images;

using the CNC data for three dimensional coordinates for the machine, the work piece, and an expected standing location of an operator assumed to be looking at the work piece for a present operation, calculating a view angle for the operator of the machine;

using the view angle, projecting the instructional information for each operation specified by the CNC data file at a point in time for completion of the operation at a location described in the data file as modified by the view angle;

using the instructional information to verify successful completion of a specified operation by comparing the properly sized and positioned images to any of the work piece and related tooling at completion of the operation; and

updating the data file and the instructional information as operations are completed.

A method of displaying guide information for a machine includes the steps of: providing a machine, the machine has at least one projectable surface, the projectable surface is located adjacent an operator workstation and adapted to reflecting a projected image;

providing a work piece, the work piece is sized, shaped and located to be used with the machine;

providing a computer numerical control (CNC) computer, the computer includes a CNC program, the program adapted to control the machine;

providing a CNC data file, the data file adapted to control performance of at least one operation by the machine upon the work piece, the data file includes positioning information for the machine and the work piece;

providing at least one calibrated projector, the projector is calibrated to dimensions of and integrated with the machine, the CNC program and the data file, the projector adapted to projecting guide information for an operator at a location where any of the work piece and related tooling is expected to be when properly positioned for a next operation, the guide information becoming visible to the machine operator when any of the work piece and related tooling is properly positioned for the next operation, the guide information includes any of textual data and properly sized and positioned images;

using the guide information to verify successful completion of a specified operation by comparing the properly sized and positioned images to any of the work piece and related tooling at completion of the operation; and

updating the data file and the guide information as operations are

completed.

(50) In a further variant, a head-mounted display system is provided. The display system provides instructional information that includes text and properly sized and positioned images and a virtual control interface reflecting actual machine controls.

(51) In still a further variant, a work piece is provided. The work piece is sized, shaped and located to be used with the machine. The CNC data file is adapted to control performance of at least one operation by the machine upon the work piece. The display system provides guide information that includes text and properly sized and positioned images appearing to be positioned upon either the work piece or related tooling. The CNC program and the data file change as operations are completed upon the work piece.

(52) In yet a further variant, the signal generating system is adapted for the machine operator to create annotations related to operation of the machine. The annotations are received by the signal receiving system and communicated to the CNC program. The CNC program saves the annotations to the CNC data file for future reference.

(53) In another variant of the invention, the signal generating system is adapted for the machine operator to create annotations related to either of the work piece or operations upon the work piece. The annotations are received by the signal receiving system and communicated to the CNC program. The CNC program saves the annotations to the CNC data file for future reference.

(54) In still another variant, the signal generating system includes a depth sensing camera. The camera generates signals describing the motions of the machine operator. (55) In yet another variant, the signal generating system includes a stereoscopic camera to generate signals describing the motions of the machine operator.

(56) In a further variant, the signal generating system includes a remote device that has an embedded motion sensor, the remote device generating three dimensional signals describing the motions of the machine operator.

(57) In still a further variant, the embedded motion sensor is selected from the group that includes gyroscopes, accelerometers and compasses.

(58) In yet a further variant, the signal generating system includes a light curtain projected on the projectable surface and a camera system detecting machine operator motions impinging on the light curtain, the camera system transmits signals describing the motions of the machine operator.

(59) In another variant of the invention, the signal generating system includes an infrared pen that contacts the projectable surface either on or adjacent to the machine and a camera system that detects machine operator motions indicated by movement of the infrared pen on the projectable surface. The camera system transmits signals describing the motions of the machine operator.

(60) In still another variant, the signal generating system further includes a pedestal. The pedestal includes a hand sensing camera. The hand sensing camera records hand motions of the machine operator and generates signals which are transmitted to the signal receiving system.

(61) In still another variant, the signal generating system further includes a head mounted hand sensing camera. The hand sensing camera records hand motions of the machine operator and generates signals which are transmitted to the signal receiving system. (62) In yet another variant, the signal generating system includes a track pad. The track pad is located either on or adjacent to the machine and detects machine operator hand motions and transmits data describing the motions of the machine operator.

(63) In a further variant, the machine further includes a safety interlock device. The interlock device requires an operator to perform a positive activity in order for the signals generated by the motion capture program to be implemented, thereby preventing accidentally initiated signals from being implemented.

(64) In still a further variant, the safety interlock device includes a light curtain. The light curtain must be engaged by an operator for the machine to be activated, thereby providing a safety interlock.

(65) In yet a further variant, the safety interlock device includes an infrared pen. A switch on the infrared pen must be engaged by an operator for the machine to be activated, thereby providing a safety interlock.

(66) In another variant of the invention, the safety interlock device includes a pressure sensor. The pressure sensor provides a signal which controls activation of the machine.

(67) In still another variant, the safety interlock device includes a strain gauge. The strain gauge provides a signal which controls activation of the machine.

(68) In yet another variant, the safety interlock device includes a control switch. The switch is manually operable by the operator.

(69) In another variant, the safety interlock device includes a switch integrally mounted with the head-mounted display system.

(70) In still another variant of the invention, the instructional information further includes a tooling location indicator. The indicator includes any of full sized images of the tooling, tooling identification, tooling size, tooling location and orientation with respect to the machine, front and side views and segmentation of the tooling at the work area at a time when the tooling is expected to be used. The tooling indicator is displayed upon any of a projectable area adjacent the work area or the tooling.

(71) In a final variant of the invention, the animation, which includes any of properly sized and positioned images of the work piece and related tooling and real time sequencing of operational steps displayed upon any of a projectable area adjacent the work area, the work piece and the tooling, is adapted for each operator of the machine by at least one view angle for the operator. The view angle is developed from the CNC data for three dimensional coordinates for the machine, the work piece, and an expected standing location of the operator assumed to be looking at the work piece for a present operation.

An appreciation of the other aims and objectives of the present invention and an understanding of it may be achieved by referring to the accompanying drawings and the detailed description of a preferred embodiment. Description of the Drawings

Figure 1 is a perspective view of a packaging machine displaying a machine operation and being controlled by an operator using hand motions;

Figure 2 is a perspective view of a press brake displaying instructional information for a work piece and tooling and being controlled by an operator using hand motions;

Figure 3 is a perspective view of the Figure 2 embodiment illustrating an animation that illustrates orientation of the work piece to the machine and movement of the work piece from a first work area to at least one subsequent work area;

Figure 4 is a perspective view of the Figure 2 embodiment illustrating a pedestal with hand motion sensing camera and safety interlock switch; Figure 5 is a perspective view of the Figure 2 embodiment illustrating a head mounted hand motion sensing camera and safety interlock switch;

Figure 6 is a perspective view of the Figure 2 embodiment illustrating

instructional and guide information for a work piece projected on the machine and on the work piece;

Figure 6A is a close-up perspective view of the Figure 2 embodiment illustrating a projection of guide information for a work piece as the work piece is moved toward the work location;

Figure 6B is a close-up perspective view of the Figure 2 embodiment illustrating a projection of guide information for a work piece with the work piece moved into the work location;

Figure 6C is a perspective view of the Figure 2 embodiment illustrating a fold-out table used for part verification with the laser projector;

Figure 6D is a close-up perspective view of the Figure 6C fold-out table supporting a work piece outlined by the laser projector for part verification;

Figure 7 is a perspective view of the Figure 2 embodiment illustrating a ring mounted signal generating system and infrared signal receiver;

Figure 8 is a perspective view of the Figure 1 embodiment illustrating annotation of the instructional information using an infrared pen signal generating system;

Figure 9 is a perspective view of the Figure 2 embodiment illustrating annotation of the instructional information using an infrared pen signal generating system;

Figure 10 is a perspective view of the Figure 2 embodiment illustrating a depth sensing camera based signal generating system;

Figure 11 is a perspective view of an infrared signal receiving system; Figure 11A is a perspective view of a hand held infrared signal generating system;

Figure 1 IB is a perspective view of the Figure 2 embodiment illustrating projected instructional information accessed by an operator with a hand held infrared signal generating system;

Figure 12 is a perspective view of the Figure 2 embodiment illustrating an infrared pen signal generating system and infrared signal receiving system;

Figure 13 is a perspective view of the Figure 2 embodiment illustrating pedestal with a hand sensing camera signal generating system and a safety interlock switch;

Figure 14 is a perspective view of the Figure 2 embodiment illustrating a stereoscopic camera based signal generating system;

Figure 15 is a perspective view of the Figure 2 embodiment illustrating a head mounted hand sensing camera and safety interlock switch;

Figure 16 is a perspective view of the Figure 1 embodiment illustrating a ring mounted signal generating system and infrared signal receiver;

Figure 17 is a perspective view of the Figure 1 embodiment illustrating a method of calibrating an optical projector to a machine;

Figure 17A is a perspective view of the Figure 1 embodiment illustrating a method of calibrating a laser projector to a machine;

Figure 18 is a perspective view of the Figure 2 embodiment illustrating a method of calibrating an optical projector to a machine;

Figure 18A is a perspective view of the Figure 2 embodiment illustrating a method of calibrating a laser projector to a machine

Figure 19 is a perspective view of the Figure 1 embodiment without a projector illustrating a camera-based signal generating and signal receiving system; Figure 20 is a perspective view of the Figure 2 embodiment without a projector illustrating a work piece for use with the machine;

Figure 21 is a perspective view of the Figure 2 embodiment without a projector illustrating a depth sensing camera signal generating and signal receiving system;

Figure 22 is a perspective view of the Figure 2 embodiment without a projector illustrating a stereoscopic camera signal generating and signal receiving system;

Figure 23 is a perspective view of the Figure 2 embodiment without a projector illustrating a ring based signal generating system, infrared signal receiving system and pressure sensor safety interlock system;

Figure 24 is a perspective view of the Figure 2 embodiment without a projector illustrating a pedestal with hand motion sensing camera signal generating system and switch operated safety interlock system;

Figure 25 is a perspective view of the Figure 2 embodiment without a projector illustrating a strain gauge safety interlock device;

Figure 26 is a perspective view of the Figure 2 embodiment with calibrated projectors illustrating a method of displaying instructional information including tooling location indicators for a machine;

Figure 26A is a close-up perspective view of the Figure 2 embodiment illustrating a tooling location indicator showing a horn tool;

Figure 26B is a perspective view of the Figure 2 embodiment illustrating an operator comparing a segmented tool to the tooling location indicator displayed on the machine;

Figure 26C is close-up perspective view of the tooling location indicator and the actual tooling mounted on the machine; Figure 27 is a perspective view of a method of developing view angles for machine operators;

Figure 28 is a perspective view of the Figure 2 embodiment without a projector illustrating an infrared pen signal generating system that contacts the projectable surface of the the machine and a camera system that detects machine operator motions indicated by movement of the infrared pen on the projectable surface;

Figure 29 is a perspective view of the Figure 1 embodiment illustrating a track pad signal generating system;

Figure 30 is a perspective view of the Figure 1 embodiment without a projector illustrating a track pad signal generating system a safety interlock switch;

Figure 31 is a perspective view of the Figure 2 embodiment illustrating a remote control with a safety interlock device;

Figure 32 is a perspective view of the Figure 2 embodiment without a projector illustrating a remote control with a safety interlock device;

Figure 33 is a perspective view of the Figure 2 embodiment without a projector illustrating an infrared pen signal generating system that contacts the projectable surface of the machine and a camera system that detects machine operator motions indicated by movement of the infrared pen on the projectable surface;

Figure 34 is a perspective view of the Figure 2 embodiment illustrating a light curtain that functions as a safety interlock device; and

Figure 35 is a perspective view of the Figure 2 embodiment illustrating head- mounted display system with integral safety interlock device. Detailed Description of the Preferred Embodiment

(1) As illustrated in Figure 1, machine display 10 providing all of the desired features can be fabricated from the following components. A machine 14 is provided. The machine 14 has at least one projectable surface 18. The projectable surface 18 is located adjacent an operator workstation 22 and adapted to reflect a projected image 26. A computer numerical control (CNC) computer 30 is provided. The computer 30 includes a CNC program 34. The program 34 is adapted to control the machine 14. A CNC data file 38 is provided. The data file 38 is adapted to control performance of at least one operation 42 by the machine 14. The data file 38 includes positioning information 46 for the machine 14. At least one calibrated projector 50 is provided. The projector 50 is calibrated to dimensions 54 of and integrated with the machine 14, the CNC program 34 and the data file 38. The projector 50 projects instructional information 58 for a machine operator 62 at a projectable location 66 on the projectable surface 18 as the instructional information 58 becomes relevant to a next machine operation 70. The instructional information 58 includes any of textual data 74 and properly sized and positioned images 78. The CNC program 34 and the data file 38 change the instructional information 58 as operations 82 are completed.

(2) In a variant of the invention, as illustrated in Figures 2, 3, 6, 6A and 6B, a work piece 86 is provided. The work piece 86 is sized, shaped and located to be used with the machine 14. The CNC data file 38 is adapted to control performance of at least one operation 82 by the machine 14 upon the work piece 86. The at least one calibrated projector 50 projects guide information 90 for operations 82 of the machine 14 directly onto any of the projectable surface 18, the work piece 86 and related tooling 94 at a time the guide information 90 is needed for the next operation 70. The CNC program 34 and the data file 38 change the guide information 90 as operations 82 are completed upon the work piece 86.

(3) In another variant, as illustrated in Figure 3, the instructional information 58 further includes an animation 98. The animation 98 illustrates orientation of the work piece 86 to the machine 14 and movement of the work piece 86 from a first work area 102 to at least one subsequent work area 106. The animation 98 includes any of properly sized and positioned images of said work piece 86 and related tooling 94 and real time sequencing of operational steps displayed upon any of a projectable area 18 adjacent the work area 102, 106 and the work piece 86 and the tooling 94. The animation 98 is linked to the CNC data file 38 and controlled by the CNC program 34. The animation 98 serves as a work flow guide 110 for a machine operator 62.

(4) In still another variant, as illustrated in Figures 6, 6A and 6B, the projector 50 projecting the guide information 90 is a laser projector 114. The laser projector 114 is adapted to project a properly scaled image 118 onto any three dimensional surface 122. (5) In yet another variant, as illustrated in Figures 6C and 6D, the guide information 90 provides direct verification of dimensions 126 of the work piece 86.

(6) In a further variant, as illustrated in Figures 1 and 16, a signal generating system 130 is provided. The signal generating system 130 generates signals 134 linked to motions 138 by the machine operator 62. A signal receiving system 142 is provided. The signal receiving system 142 receives signals 134 from the signal generating system 130 and the signals 134 are converted into gestures 146 which serve as inputs 150 to control operations 42 of the machine 14. Motions 138 by the machine operator 62 result in operations 42 by the machine 14 without physical contact between the operator 62 and any machine controls 154. (7) In still a further variant, as illustrated in Figures 2, 4, 5 and 7, a signal generating system 130 is provided. The signal generating system 130 generates signals 134 linked to motions 138 by a machine operator 62. A signal receiving system 142 is provided. The signal receiving system 142 receives signals 134 from the signal generating system 130 and the signals 134 are converted into gestures 146 which serve as inputs 150 to control operations 42 of the machine 14 upon the work piece 86. Motions 138 by the machine operator 62 result in operations 42 by the machine 14 without physical contact between the operator 62 and any machine controls 154.

(8) In yet a further variant, as illustrated in Figure 4, the signal generating system 130 further includes a pedestal 158. The pedestal 158 includes a hand sensing camera 162.

The hand sensing camera 162 records hand motions 170 of the machine operator 62 and generates signals 134 which are transmitted to the signal receiving system 142.

(9) In another variant of the invention, as illustrated in Figure 5, the signal generating system 130 further includes a head mounted hand sensing camera 174. The hand sensing camera 174 records hand motions 170 of the machine operator 62 and generates signals 134 which are transmitted to the signal receiving system 142.

(10) In still another variant, as illustrated in Figures 11 and 11 A, the signal generating system 130 includes a portable transmitter 178. The transmitter 178 is sized and shaped to be manually manipulated by a machine operator 62 and provides signals 134 indicating the position of the transmitter 178 relative to the machine 14. The signal receiving system 142 includes a receiver 182. The receiver 182 receives the signals 134 from the transmitter 178 and communicates the signals 134 to a motion capture program 186. The motion capture program 186 translates the signals 134 into gestures 146 which serve as inputs 150 which are communicated to the CNC program 34 to control operations 42 of the machine 14.

(11) In yet another variant, the signal generating system 130 includes a portable transmitter 178. The transmitter 178 is sized and shaped to be manually manipulated by a machine operator 62 and provide signals 134 indicating the position of the transmitter 178 relative to the work piece 86 and the machine 14. The signal receiving system 142 includes a receiver 182. The receiver 182 receives the signals 134 from the transmitter 178 and communicates the signals 134 to a motion capture program 186. The motion capture program 186 translates the signals 134 into gestures 146 which serve as inputs 150 which are communicated to the CNC program 34 to control operations 42 of the machine 14 upon the work piece 86.

(12) In a further variant, as illustrated in Figure 8, the signal generating system 130 is adapted for the machine operator 62 to create annotations 190 related to operation of the machine 14. The annotations 190 are received by the signal receiving system 142 and communicated to the CNC program 34. The CNC program 34 saves the annotations to the CNC data file 38 for future reference.

(13) In still a further variant, as illustrated in Figure 9, the signal generating system 130 is adapted for the machine operator 62 to create annotations 190 related to either of the work piece 86 or operations 42 upon the work piece 86. The annotations 190 are received by the signal receiving system 142 and communicated to the CNC program 34. The CNC program 34 saves the annotations 190 to the CNC data file 38 for future reference.

(14) In still a further variant, as illustrated in Figure 10, the signal generating system 130 includes a depth sensing camera 194. The camera 194 generates signals 134

describing the motions 138 of the machine operator 62. (15) In yet a further variant, as illustrated in Figure 14, the signal generating system 130 includes a stereoscopic camera 198 to generate signals 134 describing the motions 138 of the machine operator 62.

(16) In another variant of the invention, as illustrated in Figures 11 and 11 A, the signal generating system 130 includes a remote device 202 that has an embedded motion sensor

206, the remote device 202 generating three dimensional signals 134 describing the motions 138 of the machine operator 62.

(17) In another variant of the invention, the embedded motion sensor 206 is selected from the group that includes gyroscopes (not shown), accelerometers (not shown) and compasses (not shown).

(18) In still another variant, as illustrated in Figure 34, the signal generating system 130 includes a light curtain 222 projected on the projectable surface 18 and a camera system 226 detecting machine operator 62 motions 138 impinging on the light curtain 222, the camera system 226 transmits signals 134 describing the motions 138 of the machine operator 62.

(19) In yet another variant, as illustrated in Figure 12, the signal generating system 130 includes an infrared pen 230 that contacts the projectable surface 18 either on or adjacent to the machine 14 and a camera system 234 that detects machine operator 62 motions 138 indicated by movement of the infrared pen 230 on the projectable surface 18. The camera system 234 transmits signals 134 describing the motions 138 of the machine operator 62.

(20) In a further variant, as illustrated in Figure 29, the signal generating system 130 includes a track pad 238. The track pad 238 is located either on or adjacent to the machine 14 and detects machine operator 62 hand motions 170 and transmits signals 134 describing the hand motions 170 of the machine operator 62. (21) In still a further variant, as illustrated in Figures 12, 23, 25, 31 and 34, a safety interlock device 242 is provided. The interlock device 242 requires an operator 62 to perform a positive activity in order for the signals 134 generated by the signal generating system 130 to be implemented. This positive activity prevents accidentally initiated signals 134 from being implemented.

(22) In yet a further variant, as illustrated in Figure 34, the safety interlock device 242 includes a light curtain 222. The light curtain 222 must be engaged by an operator 62 for the machine 14 to be activated, thereby providing a safety interlock 242.

(23) In another variant of the invention, as illustrated in Figure 12, the safety interlock device 242 includes an infrared pen 230. A switch 246 on the infrared pen 230 must be engaged by an operator 62 for the machine 14 to be activated, thereby providing a safety interlock 242.

(24) In still another variant, as illustrated in Figure 23, the safety interlock device 242 includes a pressure sensor 250. The pressure sensor provides a signal 254 which controls activation of the machine 14.

(25) In yet another variant, as illustrated in Figure 25, the safety interlock device 242 includes a strain gauge 258. The strain gauge 258 provides a signal 262 which controls activation of the machine 14.

(26) In a further variant, as illustrated in Figures 4, 5, 13, 29 and 31, the safety interlock device 242 includes a control switch 266. The switch 266 is manually operable by the operator 62.

(27) A method, as illustrated in Figures 17A and 18A, of calibrating a machine display

10 for guide information 90 to a machine 14, includes the steps of: providing a machine 14, the machine 14 has at least one projectable surface 18, the projectable surface 18 is located adjacent an operator workstation 22 and adapted to reflecting a projected image 26; providing a computer numerical control (CNC) computer 30, the computer

30 includes a CNC program 34, the program 34 adapted to control the machine 14;

providing a CNC data file 38, the data file 38 adapted to control

performance of at least one operation 42 by the machine 14;

providing a laser projector 114, the laser projector 114 is fixedly attached either to or adjacent to the machine 14;

attaching at least three reflective targets 270 to the machine 14 at locations

274 where the laser projector 114 can reach the targets 270;

providing a three dimensional mathematical model 278 of the machine 14, the model 278 includes three dimensional coordinates 282 for each of the targets 270;

projecting a laser beam 286 at each of the targets 270 so that the beam 286 reflects directly back to the laser projector 114;

calibrating a position 290 of the laser projector 114 by comparing the three dimensional coordinates 282 of the targets 270 from the three dimensional mathematical model 278 with the positions 294 of each of the targets 270 detected by the laser beam 286;

wherein the calibration permits the laser projector 114 to project guide information 90 onto the work piece 86 and tooling 94 for the work piece 86 and an image 298 of a work piece 86 on the projectable surface 18 at a known location 302 in a designated size.

A method, as illustrated in Figures 17 and 18, of calibrating a machine display 10or instructional information 58 to a machine 14, includes the steps of:

providing at least one optical projector 306, each of the optical projectors

306 is fixedly attached either at or adjacent to a machine 14 at an identified location point 310 wherein each of the optical projectors 306 can project upon a projectable surface 18 of the machine 14; identifying a fixed reference point 314 on the projectable surface 18; projecting an alignment mark 318 on the projectable surface 18;

aligning the projected alignment mark 318 with the fixed reference point

314;

entering coordinates 322 of the alignment mark 318 into a computerized screen alignment program 326;

projecting a rectangle 330 upon the projectable surface 18;

identifying rectangular coordinates 334 of each corner 338 of the rectangle

330;

entering the rectangular coordinates 334 into the screen alignment program

326;

using the alignment program 326 to determine skew correction factors 342 for each projector 306 wherein the optical projector 306 can project a properly oriented image (not shown) upon the projectable surface 18 despite a non-orthogonal orientation to the projectable surface 18; projecting horizontal 350 and vertical 354 lines upon the projectable surface 18, measuring the horizontal 350 and vertical 354 lines;

entering the measurements 358 into the screen alignment program 326; computing horizontal 362 and vertical 366 scaling factors for each of the optical projectors 306;

wherein the optical projectors 306 will employ the screen alignment

program 326, the alignment mark 318, the skew correction factors 342, and scaling factors 362, 366 to project properly sized and positioned images 370 upon the projectable surface 18.

(29) In a variant of the method of calibrating a machine display 10 for instructional information 58 to a machine 14, further includes the steps of:

providing horizontal 374 and vertical 378 ruler projections upon the

projectable surface 18; and

providing projected measurement points 382 on the ruler projections

374, 378.

(30) In a further variant of the invention, as illustrated in Figure 19, a machine control system 12 includes a machine 14 is provided. The machine 14 has a computer numerical control (CNC) computer 30. The computer includes a CNC program 34. The program 34 is adapted to control the machine 14. A CNC data file 38 is provided. The data file 38 is adapted to control performance of at least one operation 42 by the machine 14. The data file 38 includes positioning information 46 for the machine 14. The CNC program 34 and the data file 38 change as operations 82 are completed. A signal generating system 130 is provided. The signal generating system 130 generates signals 134 linked to motions 138 by a machine operator 62. A signal receiving system 142 is provided. The signal receiving system 142 receives signals 134 from the signal generating system 130. The signals 134 are converted to gestures 146 which serve as inputs 150 to control operations 42 of the machine 14. Motions 138 by the machine operator 62 result in operations 42 by the machine 14 without physical contact between the operator 62 and any machine controls 154.

(31) In still a further variant, as illustrated in Figure 20, a work piece 86 is provided. The work piece 86 is sized, shaped and located to be used with the machine 14. The CNC data file 38 is adapted to control performance of at least one operation 82 by the machine 14 upon the work piece 86. The CNC program 34 and the data file 38 change as operations 82 are completed upon the work piece 86.

(32) In yet a further variant, as illustrated in Figure 33, the signal generating system 130 is adapted for the machine operator 62 to create annotations 190 related to operation of the machine 14. The annotations 190 are received by the signal receiving system 142 and communicated to the CNC program 34. The CNC program 34 saves the annotations to the CNC data file 38 for future reference.

(33) In another variant of the invention, as illustrated in Figure 21, the signal generating system 130 includes a depth sensing camera 194. The camera 194 generates signals 134 describing the motions 138 of the machine operator 62.

(34) In still another variant, as illustrated in Figure 22, the signal generating system 130 includes a stereoscopic camera 198 to generate signals 134 describing the motions 138 of the machine operator 62.

(35) In yet another variant, as illustrated in Figures 11, 11A and 11B, the signal generating system 130 includes a remote device 202 that has an embedded motion sensor 206, the remote device 202 generating three dimensional signals 134 describing the motions 138 of the machine operator 62.

(36) In a further variant, the embedded motion sensor 206 is selected from the group that includes gyroscopes (not shown), accelerometers (not shown) and compasses (not shown).

(37) In still a further variant, as illustrated in Figure 34, the signal generating system 130 includes a light curtain 222 projected on the projectable surface 18 and a camera system 226 detecting machine operator 62 motions 138 impinging on the light curtain 222, the camera system 226 transmits signals 134 describing the motions 138 of the machine operator 62.

(38) In yet a further variant, as illustrated in Figure 28, the signal generating system 130 includes an infrared pen 230 that contacts the projectable surface 18 either on or adjacent to the machine 14 and a camera system 234 that detects machine operator 62 motions 138 indicated by movement of the infrared pen 230 on the projectable surface 18. The camera system 234 transmits signals 134 describing the motions 138 of the machine operator 62.

(39) In another variant of the invention, as illustrated in Figures 13, 24 and 30, the signal generating system 130 further includes a pedestal 158. The pedestal 158 includes a hand sensing camera 162. The hand sensing camera 162 records hand motions 170 of the machine operator 62 and generates signals 134 which are transmitted to the signal receiving system 142.

(40) In still another variant, as illustrated in Figure 15, the signal generating system 130 further includes a head mounted hand sensing camera 174. The hand sensing camera 174 records hand motions 170 of the machine operator 62 and generates signals 134 which are transmitted to the signal receiving system 142.

(41) In yet another variant, as illustrated in Figure 29, the signal generating system 130 includes a track pad 238. The track pad 238 is located either on or adjacent to the machine 14 and detects machine operator 62 hand motions 170 and transmits signals 134 describing the hand motions 170 of the machine operator 62.

(42) In a further variant, as illustrated in Figures 12, 23, 25, 31 and 34, the machine 14 further includes a safety interlock device 242. The interlock device 242 requires an operator 62 to perform a positive activity in order for the signals 134 generated by the signal generating system 130 to be implemented. This positive activity prevents accidentally initiated signals 134 from being implemented.

(43) In still a further variant, as illustrated in Figure 34, the safety interlock device 242 includes a light curtain 222. The light curtain 222 must be engaged by an operator 62 for the machine 14to be activated, thereby providing a safety interlock 242.

(44) In yet a further variant, as illustrated in Figure 12, the safety interlock device 242 includes an infrared pen 230. A switch 246 on the infrared pen 230 must be engaged by an operator 62 for the machine 14 to be activated, thereby providing a safety interlock 242.

(45) In another variant of the invention, as illustrated in Figure 23, the safety interlock device 242 includes a pressure sensor 250. The pressure sensor provides a signal 254 which controls activation of the machine 14.

(46) In still another variant, as illustrated in Figure 25, the safety interlock device 242 includes a strain gauge 258. The strain gauge 258 provides a signal 262 which controls activation of the machine 14. (47) In yet another variant, as illustrated in Figures 24 and 30, the safety interlock device 242 includes a control switch 266. The switch 266 is manually operable by the operator 62.

(48) A method, as illustrated in Figures 1, 2, 6, 6A, 6B and 27, of displaying

instructional information 58 for a machine 14 includes the steps of:

providing a machine 14, the machine 14 has at least one projectable surface

18, the projectable surface 18 is located adjacent an operator workstation 22 and adapted to reflecting a projected image 26; providing a work piece 86, the work piece 86 is sized, shaped and located to be used with the machine 14;

providing a computer numerical control (CNC) computer 30, the computer

30 includes a CNC program 34, the program 34 adapted to control the machine 14;

providing a CNC data file 38, the data file 38 adapted to control

performance of at least one operation 42 by the machine 14 upon the work piece 86, the data file 38 includes positioning information 46 for the machine 14 and the work piece 18;

providing at least one calibrated projector 50, the projector 50 is calibrated to dimensions 54 of and integrated with the machine 14, the CNC program 34 and the data file 38, the projector 50 adapted to projecting instructional information 58 for an operator 62 at a projectable location 66 on the projectable surface 18 as the instructional information 58 becomes relevant to a next machine operation 70, the instructional information 58 including any of textual data 74 and properly sized and positioned images 78;

using the CNC data 38 for three dimensional coordinates 386 for the

machine 14, the work piece 86 , and an expected standing location 390 of an operator 62 assumed to be looking at the work piece 86 for a present operation 394, calculating a view angle 398 for the operator 62 of the machine 14;

using the view angle 398, projecting the instructional information 58 for each operation 42 specified by the CNC data file 38 at a point in time for completion of the operation 42 at a location 402 described in the data file 38 as modified by the view angle 398; using the instructional information 58 to verify successful completion of a specified operation 42 by comparing the properly sized and positioned images 78 to any of the work piece 86 and related tooling 94 at completion of the operation 42; and

updating the data file 38 and the instructional information 58 as operations

42 are completed.

A method, as illustrated in Figures 1, 2, 6, 6A, 6B and 26, of displaying guide information 90 for a machine 14 includes the steps of:

providing a machine 14, the machine 14 has at least one projectable surface

18, the projectable surface 18 is located adjacent an operator workstation 22 and adapted to reflecting a projected image 26; providing a work piece 86, the work piece 86 is sized, shaped and located to be used with the machine 14; providing a computer numerical control (CNC) computer 30, the computer 30 includes a CNC program 34, the program 34 adapted to control the machine 14;

providing a CNC data file 38, the data file 38 adapted to control

performance of at least one operation 42 by the machine 14 upon the work piece 86, the data file 38 includes positioning information 46 for the machine 14 and the work piece 18;

providing at least one calibrated projector 50, the projector 50 is calibrated to dimensions 54 of and integrated with the machine 14, the CNC program 34 and the data file 38, the projector 50 adapted to projecting guide information 90 for an operator 62 at a location 406 where any of the work piece 86 and related tooling 94 is expected to be when properly positioned for a next operation 410, the guide information 90 becoming visible to thejnachine operator 62 when any of the work piece 86 and related tooling 94 is properly positioned for the next operation 410, the guide information 90 includes any of textual data 74 and properly sized and positioned images 78;

using the guide information 90 to verify successful completion of a

specified operation 42 by comparing the properly sized and positioned images 78 to any of the work piece 86 and related tooling 94 at completion of the operation 42; and

updating the data file 38 and the guide information 90 as operations 42 are completed. (50) In a further variant, a head-mounted display system 414 is provided. The display system 414 provides instructional information 58 that includes text 74 and properly sized and positioned images 78 and a virtual control interface 418 reflecting actual machine controls 422.

(51) In still a further variant, as illustrated in Figures 2, 3, 6, 6A and 6B, a work piece 86 is provided. The work piece 86 is sized, shaped and located to be used with the machine 14. The CNC data file 38 is adapted to control performance of at least one operation 82 by the machine 14 upon the work piece 86. The display system 414 provides guide information 90 that includes text 74 and properly sized and positioned images 78 appearing to be positioned upon either the work piece 86 or related tooling 94. The CNC program 34 and the data file 38 change as operations 82 are completed upon the work piece 86.

(52) In yet a further variant, as illustrated in Figures 1 and 16, the signal generating system 130 is adapted for the machine operator 62 to create annotations 190 related to operation of the machine 14. The annotations 190 are received by the signal receiving system 142 and communicated to the CNC program 34. The CNC program 34 saves the annotations to the CNC data file 38 for future reference.

(53) In another variant of the invention, as illustrated in Figures 2, 4, 5 and 7, the signal generating system 130 is adapted for the machine operator 62 to create annotations 190 related to either of the work piece 86 or operations 42 upon the work piece 86. The annotations 190 are received by the signal receiving system 142 and communicated to the CNC program 34. The CNC program 34 saves the annotations 190 to the CNC data file 38 for future reference. (54) In still another variant, as illustrated in Figure 10, the signal generating system 130 includes a depth sensing camera 194. The camera 194 generates signals 134 describing the motions 138 of the machine operator 62.

(55) In yet another variant, the signal generating system 130 includes a stereoscopic camera 198 to generate signals 134 describing the motions 138 of the machine operator 62.

(56) In a further variant, as illustrated in Figures 11, 11A, 11B and 32, the signal generating system 130 includes a remote device 202 that has an embedded motion sensor 206, the remote device 202 generating three dimensional signals 134 describing the motions 138 of the machine operator 62.

(57) In still a further variant, the embedded motion sensor 206 is selected from the group that includes gyroscopes (not shown), accelerometers (not shown) and compasses (not shown).

(58) In yet a further variant, as illustrated in Figure 34, the signal generating system 130 includes a light curtain 222 projected on the projectable surface 18 and a camera system 226 detecting machine operator 62 motions 138 impinging on the light curtain 222, the camera system 226 transmits signals 134 describing the motions 138 of the machine operator 62.

(59) In another variant of the invention, as illustrated in Figure 12 and 33, the signal generating system 130 includes an infrared pen 230 that contacts the projectable surface 18 either on or adjacent to the machine 14 and a camera system 234 that detects machine operator 62 motions 138 indicated by movement of the infrared pen 230 on the projectable surface 18. The camera system 234 transmits signals 134 describing the motions 138 of the machine operator 62. (60) In still another variant, as illustrated in Figure 24, the signal generating system 130 further includes a pedestal 158. The pedestal 158 includes a hand sensing camera 162. The hand sensing camera 162 records hand motions 170 of the machine operator 62 and generates signals 134 which are transmitted to the signal receiving system 142.

(61) In still another variant, as illustrated in Figure 5, the signal generating system 130 further includes a head mounted hand sensing camera 174. The hand sensing camera 174 records hand motions 170 of the machine operator 62 and generates signals 134 which are transmitted to the signal receiving system 142.

(62) In yet another variant, as illustrated in Figure 29, the signal generating system 130 includes a track pad 238. The track pad 238 is located either on or adjacent to the machine

14 and detects machine operator 62 hand motions 170 and transmits signals 134 describing the hand motions 170 of the machine operator 62.

(63) In a further variant, as illustrated in Figures 12, 23, 25, 31 and 34, the machine 14 further includes a safety interlock device 242. The interlock device 242 requires an operator 62 to perform a positive activity in order for the signals 134 generated by the signal generating system 130 to be implemented. This positive activity prevents accidentally initiated signals 134 from being implemented.

(64) In still a further variant, as illustrated in Figure 34, the safety interlock device 242 includes a light curtain 222. The light curtain 222 must be engaged by an operator 62 for the machine 14 to be activated, thereby providing a safety interlock 242.

(65) In yet a further variant, as illustrated in Figure 12, the safety interlock device 242 includes an infrared pen 230. A switch 246 on the infrared pen 230 must be engaged by an operator 62 for the machine 14 to be activated, thereby providing a safety interlock 242. (66) In another variant of the invention, as illustrated in Figure 23, the safety interlock device 242 includes a pressure sensor 250. The pressure sensor provides a signal 254 which controls activation of the machine 14.

(67) In still another variant, as illustrated in Figure 25, the safety interlock device 242 includes a strain gauge 258. The strain gauge 258 provides a signal 262 which controls activation of the machine 14.

(68) In yet another variant, as illustrated in Figures 24, 30 and 31, the safety interlock device 242 includes a control switch 266. The switch 266 is manually operable by the operator 62.

(69) In another variant, as illustrated in Figure 35, the safety interlock device 242 includes a switch 426 attached to the head-mounted display system 414.

(70) In still another variant, as illustrated in Figures 26, 26A, 26B and 26C, the instructional information 58 further includes a tooling location indicator 430. The indicator 430 includes any of properly sized and positioned images 436 of the tooling 94, tooling identification 440, tooling size 444, tooling location 448 and orientation with respect to the machine 452, front 456 and side 460 views and segmentation 464 of the tooling 94 at the work area 468 at a time when the tooling 94 is expected to be used. The tooling indicator 430 is displayed upon any of a projectable area 18 adjacent the work area 468 or the tooling 94.

(71) In a final variant of the invention, as illustrated in Figures 2, 3 and 27, the animation 98, which includes any of properly sized and positioned images of the work piece 86 and related tooling 94 and real time sequencing of operational steps displayed upon any of a projectable area 18 adjacent the work area 102, 106, the work piece 86 and the tooling 94, is adapted for each operator 62 of the machine 14 by at least one view angle 398 for the operator 62. The view angle 398 is developed from the CNC data 38 for three dimensional coordinates 386 for the machine 14, the work piece 86, and an expected standing location 390 of the operator 62 assumed to be looking at the work piece 86 for a present operation 394.

The machine display 10 and related control systems 12 and safety interlock devices

242, related methods of calibrating projectors 50 for said machines 14 and methods of displaying instructional 58 and guide information 90 for use with said machines 14 have been described with reference to particular embodiments. Other modifications and enhancements can be made without departing from the spirit and scope of the claims that follow.