ME T HOD S FOR CONTROLLING EQUIPM EN T AM) CON TROLLE D

EQUIPMENT USING INVERSE U ER MODELS

CROSS-REFERENCE TO RELATED APPL!CATIONCS)

1 0.1 J litis application claims the. benefit under 35 U.S.C. § 119 of the earlier filing date of U.S. Provisional Application. Serial No. 62/210,886 filed August 27, 2015. the entire contends o which, are hereby incorporated by reference in their en tiret for any purpose. This. instant application, also claims the benefit under 35 U.S.C. § 11 of the •earl er filing date of U.S. Provisional Application Serial No. 62/323.434 filed April IS, 2016, the entire contents of which are hereby Incorporated by reference in their entirety for any purpose.

TECH NIC AL FI ELD

PNJ2J Embodiments of the !nve ton relate generally to control of robotics, machinery of other equipment, and examples control based on models of hutuan activity are described. Examples of achieving a trajectory of a robotic or other equipment component are described which may infer intended trajectory tk m. human input.

BACKGROUND

[003 j Development of technologies geared toward easier operation by novice human users has been desired, e.g. using back-driveable -robots where gravity compensation is provided, along with impedance modification, so that the user can teach by physically moving the robot end-effector by hand, rather than programming the robot to do specific tasks. Other applications include situations where a machine needs to be trained to follow a desired trajectory in a repetitive scenario, e.g. pick and place operations in a ^' matia&c uririg setting, or potentially training an acti ve prosthesis during a walking cycle.

|βθ4| I the user is an expert at a task, then Learning, from Demonstration (LiD) approaches may be available to use- human demonstrations to train a machine. For example, the observed movement can be represented using Dynamic Movement Primitives (DMPs), which can then be used, to reproduce the . movement.. Although this is a ooeshot learning approach; that can be generalized to slightly varying situations. DMPs inherently rely n the quality of the initial demonstration from an expert. Another approach is Inverse Remibreement Learning, which can. be: used to learn a policy without an explicit reward function, but again relies on. demonstrations by an expert. Thus, the desired output trajectory, that ^' is initially not known, can he learnt through human demonstrations, if th human user is an expert at achieving the desired output tracking. Such methods are not designed to do better than the initial demonstration for the same task, and axe therefore, not suited for .situations- when the human user cannot achieve the desired precision in output tracking.

051 If the desired output, y ₍ j is directl -available to the coMtoller, then traditional Iterative Learning Control (ILC) can be used to improve the output ^' tracking performance based on the tracking error, e ^∞ a - y. Conditions tor convergence of such iterative algorithms have been well studied. In the context of LIT), reinforcement learning using reward feedback (based on the tracking error, e) has been studied. Knowledge: of the desired out ut \¾ is also needed, in- Feedback- Error Learning (FEL) that uses- neural, networks for human-like learning. However, such approaches ^' axe not applicable to situations where the novice human user night not have a direct method to describe the desired output y _< j to the controller, and therefore, the, desired output y _<5 and the tracking error e are not available to the controller.

SUMMARY

[006| Examples of computing systems are described herein. In some examples, a computing system, includes at least one processing unit and at least one computer readable media encoded with executable instructions that, when executed, cause the computing system to perform, actions including receive user input for controlled equipment, calculate adjusted input based on. he user input and using a stored model of a feedback transfer function between user input and trajectory error, and provide the adjusted input to the controlled equipment

[W7J In some examples, using the stored model of the feedback transfer function comprises using an inverse model of the feedback transfer function.

fee81 in some examples, the stored model of the feedback transfer function comprises data determined through empirical observation of user interaction with the controlled equipment or simitar controlled equipment. j ⁽ Mi9) In some examples, the adjusted input includes a. portion- equal to a product of a previous user mput: and an inverse of the stored model of the feedback transfer iimction,

|Mt}{ In some examples, the adjusted input includes a portion equal to a product of a previous user input, an inverse of the stored model of the feedback transfer function, and a gai In some examples, the gain is selected to provide convergence on a desired trajectory.

[011} In some example*, the computing system may be configured to perform online inverse computation of the controlled system. In some examples, the controlled equipment is higher-order system, a id the online inverse computation provides an apparent system including fee control led system behaving as a lower-order system, ft>!2{ In some examples, the computing system may he configured to perform preview-based online inversion of the controlled system. In some examples, the preview-based online- inversion uses art approximation of an unstable subs stem portion of the Controlled system. In some examples, the -computing system may be configured to fi lter the user input using a cascade of low-pass filters.

[013} Examples of methods are described herein. In some examples, a method includes receiving user input for controlled -equipment, ^' calculating adjusted inputs for the controlled equipment based on a model of a feedback transfer function, and applying the adjusted inputs to the controlled e uipment.

$1 } In some examples, receiving user input comprises receiving a set of user input corresponding to a user attempting to achiev a desired -trajectory with the controlled equipment,

[015} In some examples, applying the adjusted inputs occurs after receiving the set of user input, and the set of user input fails to achieve the desired trajectory and the adjusted inputs achieves closer to the desired trajectory.

[0.½} In. some examples, applying the adjusted inputs occurs during a time further user input is being received tor the controlled equipment,

[017} In some examples, calculating adjusted inputs is based on an inverse of the model of the feedback transfer function. in some examples, methods may nclude determining the mode! of the feedback ransfer function from ^' records of previous interaction between users and the . ^. controlled equipment,

^' [0ΐ9[ in some e am les, methods .may include performing online inversion to generat ^' an apparent system including the controlled equipment. In some examples, the apparent system is a Sower-order system than the controlled equi me t. In s me examples, the onl ne inversion com rise providing an inverse model of the controlled equipment.

[826} In some examples, methods niay include performing preview-based online inversion to generate an apparent system including th controlled .equipment, in some examples, methods ma include filtering the user input using a cascade of low-pass filters.

[02 If In some examples, the user may comprise an autonomous agent and the feedback transfer function may be associated with a. neighboring agent in a network,

BRIEF DESCRIPTION ^' OF THE RA WING S

{022} FIG. .1 is a schematic illustration of a system arranged in accordance with examples described herein.

j 3j FIG. 2 is a schematic illustration of a method arran ^' ed in accordance with examples described herein.

[0241 FIG. 3 is a schematic illustration of a system for providing input to controlled equipment arranged in accordance with examples described herein.

[02S| .FIG. 4 is a schematic illustration of a system for providing input to controlled equipment arranged in accordance with, examples described herein.

{ ^' 026} FIG. 5 i a block diagram of online inverse computation system for an example system G.

[02?f FIG. 6 is a schematic illustration of an online inversion of a controlled system using a preview-based online inversion technique,

[928| FIG. 7 is a schematic illustration of a cascade of two first-order low-pass filters which may be used to find a filtered signal and. its time derivatives.

[029} FIGS. 8A~D are graphs depicting output of an example ir»ple.f»entaiio.« of sy stems described herein. DETAILED DESCRIPTION

[030] Geriain details are Set fort below to provide: a sufficient understanding of embodiments of the invention. However, It will be clear to one, skilled in the art that embodiments of the invention ma be practiced without various of these particular details, in some instances, well-known circuits, control signals, timing protocols, and software operations have -not been shown - is detail ro order to avoid unne essaril obscuring the described embodiments of the invention,

[031} Examples described herein may pertain to human-guided iterative learning control for -output-trajectory tracking tasks, whet? the controller may not have direct access t the desired output and needs to infer it from human demonstrations, it may he challenging to develop such demonstration based learning in situations where the novice human user might not be. able to achieve precision tracking. Examples described herein .include an inversion-based approach to infer the human intent and use it to design the input update for improving the Output tracking. In some examples, conditions are developed for convergence of the proposed iterative output-tracking control for linear time-invariant systems.

[032 j Examples described herein may utilize an inversion-based approach to infer human intent (e.g., what trajectory the user is trying to achieve) and use it to select the input update fo improving output tracking. This may be achieved in some examples even if the user does not have sufficient expertise to achieve output tracking with the machine. In some examples, the inversion-based approach may be used to infer inten of autonomous agents in a network, which may improve performance of the entire network. For example, a user may be an autonomous agent and inferred intent of one or more neighboring agents in a. network, may be used to provide an input update.

[t ⁾ 33| Figure .1 i a schematic illustratio of a system arranged in accordance with examples described herein. The system 1 14 of Figure 1 includes user ^■ input 102, computing system 104, processin unit(s) 106, nxxfe!(s) of feedback transfer function 1 16, executable instructions for adjustment .108. adjusted input 110, and. controlled equipment .1 12. Generally, during operation,, the user input 102 is provided to the computing system 104 which may .calculate the-adjusted input. 110 using the processing «nit(s) 106 and executable instructions for adjustment 108. The adjusted: input 1 .10 is then provided to controlled equipment 112 to control the controlled equipment 1 12. jf>34) Any of a variety of equipment may be used to implement the controlled equipment 1 12. That is, any of a variety of equipment may be .controlled using systems and methods described herein. Examples, ^" include, but are not limited to, physical therapy robotic machines, robotic systems* construction equipment such as backhoes OF torklifts, transportation equi ment such as automobiles, airplanes, trains, or buses, or manufacturing ^' equipment

{835} in some examples described herein, a. user may desire that . a portion of the controlled equipment move in a particular trajectory. Accordingly, in some examples, a xise.r ma provide input, e.g. use input 102, to move the. portion of the controlled equipment in the particular trajectory. The user may, for example, manually move a portion of the controlled equipment 12 (e.g. move a pedal, wheel, arm, end-effector, or other portion of the controlled equipment) in the manner the user desires of the equipment to move when controlled robotic-ally. The manual motion may be- the user input 102, in some: examples, the user may operate a..user ^' interface, to the -controlled equipment 1 12 to provide the user ^■ input 102 for example, the user may operate a joystick, gas. pedal, brake pedal, throttle, steerin wheel, may gesture in front of a. camera or other gesture-recognition device, ma use a graphical user interface (e.g. such as a Microsoft Kineet sensor), may touch a touch, screen interface, or ma operate Other user interlace devices- to provide the user input .1 2.

{036} In providing the use input 102, the user may not achieve exactly the intended trajectory. For example, when manually moving a portion of the controlled equipment H-2 and/or when operatin a user interface to the controlled equipment 1 12, the user may {perhap uu eftiioiially} deviate from, the desired trajectory, if the user, - for example, desires for a portion of the controlled machinery to move in a circular trajectory, the user input 102 may result in ^' a precise circular trajectory, but may be a somewhat different trajectory given, for example., the vagaries of human, .input and/or perhap error or inexperience On the part of the error. Any of a variety o desired: trajectories may be desired by the user, e.g. pick-and-place operations for manufacturing equipment, a. walking cycle for prosthetic equipment, or others.

[837} Accordingly, the computing system. 10 may adjust the user .input 102 to provide adjusted input 110 fo .the ^' Controlled equipment 1 12 to better achieve, a desired trajectory . Of course, in some examples, th desired trajectory may not be known to the computing system 104. In some examples, the desired trajectory may he inferred by the computing system 104 from the .user, input 102 arid the ^' adjusted input 110 may be provided: that achieves the desired trajectory . The adjusted input 110 may be provided to control the controlled equipment 132 - for example the adjusted input ! 30 may be applied as control signals to various components of the controlled equipment 1 12 that may be moved. In some examples, the adjusted input 110 may he used to control user interface devices used to operate the controlled equipment 1 12. n some examples, the adjusted input 1 10 may foe added to or used ^' to augment the user input 102 in controlling the controlled equipment 1 1 ,

0381 Ϊ» some examples, the user input Stay he human .input. In some examples, the user may be as autonomous agent (e.g. another piece of equipment, a. software process, or other automated actor). For example, an autonomous agent in a network may pro vide user input to a neighboring autonomous .agent in the network,

839} The com utin system 164 mm meio.de one. or more processing imit(s) 106 (e.g. one or more processors and/or controllers) and memory (e.g. RAM, ROM., hard disk, SSD, or other memory) encoding executable instructions for adjustment .108. The computing system 104 may include a variety of input and/or output devices including but not. limited to a touchscreen, mouse, keyboard, camera, speaker, microphone, or combinations thereof The computing system 104 may be implemented using a server., computer, desktop, laptop, tablet, cellular phone, and/or in some examples the computing system 104 may be integrated with the controlled equipment 1 12. For example, the ..controlled equipment 1 12 may itself include processing umtis) 106 and/or executable instructions for adjustment 108 used to generate the adjusted input 110. in some example, the computing system. 10 may be implemented using hardware and/or firmware, such as fPGAs or other configurable circuitry that acts as the processing unit ^' s) 106 and executable instructions tor adjustment 1 8.

840) In. providing the adjusted input. 110, the computing system 104 may utilize model(s) of feedback transfer function 116, The model(s) of feedback transfer function 1 16 may be stored in electronic storage integrated with and/or accessible to the computing system 104. The: modelis) of feedback transfer function 116 may be modeled, relationships between a user input to the controlled equipment 112 and an error between a desired and actual trajectory. The model(s) of feedback transfer function .116 may include modei(s) of a feedback transfer function between user input and trajectory error, The model(s) of feedback transfer function ! 16 may be stored as that relationship, or may he stored in a ions indicative of that, relationship in a mathematical way. The modeiis) of feedback transfer function. 1 16 ma be stored as the forward or inverse model. I some, examples, the computing system 104 may perform me inverse of the model The .modei(s of feedback transfer function i.1.6 may be empirically determined - e.g. they may be developed by observation of user interaction with the controlled equipment 112 or similar equipment or in making similar traj ectories with other equipment The ^' modeiis) of .feedback transfer function 1 16 may be estimated or mathematically or theoretically proposed based on human dynamics, machine dynamics, or other factors. An number of modeiis) of feedback transfer function. 116 may be stored. The modei(s) .of feedback transfer function. 16 may be stored specific to particular users: or types: of users providing user input 102■ and/or specific to particular controlled equipment 1 12 or types of controlled equipment 1 12. Accordingly, in some .examples, the. Computing system 104 may select a model from the modeiis) of feedback, transfer function i 16 to use based on the user, type of user, controlled equipment, type of controlled equipment, type of action to he performed, or combinations thereof

| In some examples, the model(s) of feedback transfer function may be models of human activity (e.g. a model of human action based on intended trajectory). In some examples, the modeiis) of feedback, transfer function may be models of autonomous agent activity (e .g. a model of autonomous agent actio based on. i ntended trajectory _. }.] A variety of models may be stored for use in the ^' modeifsj of feedback transfer function 1 16. In some examples, an analytical-verbal model, may be used. Analytical- verbal models may be used in some examples to model a human response for a manual control, task. The analytical-verbal model may be used when the controlled equipment 112 is approximated as a second-order system. Other models or techniques, such as neural, networks, may be used to provide the modeiis) of feedback transfer function 1 16 in some examples. The analytical-verbal model may be based on a crossover model, for example a crossover model of human feedback response: to single-loop compensatory tracking tasks, such as the openloop crossover model Other models may also be used, including but not limited to, driver models such as optimal control models, P control models, fuzay logic models, and/or Markov models. Models based •on algorithmic, representations may be used i some examples, such as the Feedback Error Learning model or ARX models with updated model parameters _, based on fuzzy inference.

|ft43| In some examples, user-response models (e.g. human-response models, such as model(s) of feedback transfer function 116 ) may only be available for limited types of control-led systems (e.g. -limited types of controlled equipment 1.12). Some example systems and methods described herein may infer user intent for more complex systems (e.g. higher than 2nd order systems), by using an ^' inversion-based approach So specify that the apparent system perceived by the: user is one for which user-response models are known- to exist. For example, online inversion computation may be used, in some examples, an apparent system provided by the inversion-based approach may be a lower-order system (e.g. 2nd order or lower) and the controlled system may be a higher«order system (e.g. 3rd order or higher). Then, the inferred- intent (e.g. output trajectory- found by inverting the human-response model) may be used to iteratively find precision output-tracking inputs for the controlled equipment 1 12 which itself may .not have a associated models) of feedback transfer function 1 16, _. but which may be modeled as -a system, which does.

IM4\ In some examples, where a complex system may no have a stable inverse, a preview-based approach may be osed to provide an apparent system that may be used together with a known modei(s) of feedback transfer function 116,

[045] Figure 2 is a schematic illustration f a method arranged in accordance with examples described herein. The method 200 includes block 202, block 204, and block. 206. Block 202 recites "receive user input" and m y be folio wed by block 204 which recites "calculate adjusted input" which may be followed by block 206 which recites "operate controlled equipment using adjusted tuput". It- is to be understood that, in some examples the various blocks shown in Figure.2 may be rearranged and/or may occur in part simultaneously. For example, in some examples, the calculation, of the adjusted input in block 204 may occur dynamically while additional user input from block 202 is. being received. In. some examples, a complete set of user input may be provided in block 202 prio to calculating the. adjusted input in block 204. The computing system instructions for adjustment 10.8 may include instructions for calculating the adjusted input in block 204. The comp t ng system 104 of Figure 1 may operate the■. controlled equipment using adj usted input: in block 20:6.

t 6j The user input 102 may be received in block 202, for example by receipt of movements made by a user either of portions of the -controlled equipment or signals provided by a user interface to the controlled equipment, for example, h : some examples, the user input received in block 202 may include a set of user input corresponding to a user attempting to achieve a desired trajectory with the controlled equipment,

947} In block 204 it ad justed input may be calculated, in examples described he ei ;, the adjusted input may be calculated using in version of previously established models of human-in-th.e-i.oop behavior to estimate the user intent. For example, a model of a feedback transfer function ma b used. The model may be inverted for use in calculating the adjusted input in block; 204 - .e.g. the inverted feedback transfer function may be used. In block 206, the controlled equipment may be operated using the adjusted input. For example, the adjusted, input may be provided to the controlled equipment, in some examples, the model of the feedback transfer .function may be determined (e.g. by the computing, system 104 of Figure 1 or another computing system) from records of previou interaction between, users and the controlled • equipment or simitar controlled equipment. The .model determination may occu prior to receip of user input in block 202 in some examples, for example during a calibration procedure.

948} in some examples, applying the adjusted inputs i block 206 to the controlled equipment may occur after receiving a set of user input corresponding to a user attempting to achieve a desired trajectory with the controlled equipment, m some examples, ihe set of user input may fail, to achieve the desired trajectory and the adjusted inputs .may achieve closer to the desired trajectory.

M9) In some examples, applying the adjusted inputs in block 20 ma occur during a time farther user input is being received for the controlled equipment in block 202. For example, the adjusted input in block: 204 may be calculated dynamically during operation of the controlled equipment b a user such that user input to the controlled. equipment is augmented with the adjusted inputs to improve the operation of die ■controlled eq ipment

[dSffj Figure 3 is a schematic illustration of a system for providing input to. controlled equipment arranged in accordance with examples described herein The system includes combiner 302, ^' user response 304, user feedback response 306, user machine interface 308, machine controller 310, combiner 31:2, and controlled system 314, The desired output signal is represented in Figure 3 as \¾. That, desired output signal (e.g. Desired trajectory) may be known only to a user of the system, which may be a human user. An actual output of the controlled system 314 (which may be implemented, for example, by any controlled equipment described herein, such as the controlled, equipment 1 12 of Figure 1 } is represented as y . There is an error, shown as e in Figure 3, between the desired output. _\¼ and the actual output y. The error is provided by the combiner 302 which subtracts the actual output y from the desired output to yield the error, e. For example, ¾ - y ^™ e,. The error may prompt a user response 304, shown as ½ in Figure 3» The user response 304 may include two components - a user feedback response 306, gi ven as H¾ in Figure 3, and a user machine interface 308 given as K¾ in Figure 3. The user feedback response 306 may be implemented using measured f equency response data irt some examples, f or example, during a. training process, a user: may provide inputs responsive to a variety of err rs e, and. the measured frequency response data may be used to provide i¼. In some examples, nominal parameter values may be used for the human feedback response .model ¾,, The nominal parameter values may come from prior observed ^' results or from knowledge of how the human reaction takes place.

[051 j Generally,, the user feedback response 306 may refer to the action user may ake responsive to the input e. The action may be to move a portion of the controlled system 314 and/or to fake an action in or on a use -.interface, to the. controlled system 31.4., The user machine -interface 308 t may generally refer to the correlation between the- user's action and a signal or other direct input to the controlled system 314. In this manner, the user's action may result in provided response « _¾ in light of the error e. The model between e and the user response u¾ is the- human response Gj-j,

[052J. Examples described herein may provide a machine controller- 310 which may calculate an adjustment to the response u¾, the adjustment may be shown in Figure 3 as ii _c . The machine cosiiroikr is shown in Figure: 3 as implementing a function. G* ^' between the user response «¾ and die adjustment The. machine controller 310 may be implemented, for example using the computing system 104 of Figure K The combiner 3 2 may combine the adjustment ¾ and the user response t% such as b adding the responses, to provide an adjusted input u for the controlled system 31 . Given the input u _¾ the controlled system 314, which ma be modeled as the function G shown, in. Figure 3, may provide an output y. The output y may be fed back -and. compared at combiner 302 with the desired trajectory s¾ resuiting in error ε, and the process continues.

|¾S3f Examples described herein aim to achieve the desired output trajectory, y ^»» for a system, G using a shared control strategy as depicted in F igure 3 where both the controller, G _c and the human response model, On (which may model the human input to the perceived error e) provide inputs and %, respectively. The- resulting output, y, may -be described in the frequenc domain by

.Equation 1

where the output y is a product of the user response 304 mode! and the input to the controlled system 314, u. Where the input to the controlled system u is the sum of the user provided input Uh and the controller input u _c . That can be considered, as shown in Equation 1, to he equal to a feedback transfer .function G; _¾ multiplied by the output y i summed with a feedforward transfer function C½ multiplied by the controller input Uc, where the feedback,. G¾ and feedforward, Gn ^; transfer functions are given by

Equations 2 & . 3

j SSj respectively, all expressed as frequency response functions, for example where G(m) z= G(s) , where j = *< —t .

Equation

$56} In the above Bqisatkms 2 and 3 the user response model G« may be give as

Equation 5

[057] where Kl represents the user-maehiae interface and H _t¾ represents he compensatory feedback that aims to correct the perceived error, e. Note tha the model of the human response model, (¾ relates the feedback, G _¾ and feedforward, G _ff transfer functions in Equations 2 and 3 as

Equation 6

in theory, the user iapu ^' t % may include other terms such as a feedforward component t¾ (w)y (w) that, may be directly related to the desired output y$. However, such effects may be small at low frequencies., and may be neglected in examples described herein

jfiS9] Examples described herein may provide the eoniroller input, t¾, which may be referred to as an adjustmen or, in some examples, an adjusted input, ^' that achieves output tracking of a desired output trajectory, y _t j under one or both of the following two constraints.

|8i¾f Constraint 1 refers to when, the controller, Ge does not have direct access to the desired output trajectory, }¾, hut, may have direct access to the control input from the user, u;..

|iJ6if Constraint 2 refers to whe the controller, Gc does not. have exact information ^; of the feedback G,¾ transfer function, hut has access to a model of (¾>, notated as G ¾. j§62) The output y at the k ^f " iteration -can be expressed as:

}¾ (m) = C? ¾ ( m )y _£ i ( co ) +€7 f ( ) u _c ^ { ⁽ $)*

E uation 7

jl½3 j where e k ^f¾ iteration of the output y is equal to the product, of the feedback transfer ftmctian. G _¾ and the desired output y,j plus the product of the. feedforward transfer unctiatrGfl' and the k* machine controHer input.U _c .

|86 } Subtracting desired output y _ti from both sides of Equation ?, the error, can be expressed, as:

Equation 8

[MSf where the k* error equals the desired output minus the actual k* .output yu.

That expression can be written as the product of one minu the feedback transfer f nction G _f t _> with the desired output v¾. subtracting the product of the feedforward transfer function G and the k ^lh controller input An analogous expression can be written for the k-M^ error, and subtracting the two yields a relationship between two consecuti ve errors between updates: '*·♦· i (» e _k (m) ~~ G _f f(m) Ui _ik+ i im) ~~ tt _Ji ^{ )l

Equation 9 ^'

|β66} where the k-H* error is the error minus the product of the feedforward transfer function 0» with the difference between the controller input at consecutive iterations, ¾Α _* ί and u _c . _k . j ^§ 67) Generally,- in examples described hernia, ft may be desirable to reduce the tracking etror (e,g, e¾) to 0 as the iterations progress. Reviewing Equation 9, the tacking error ¾·Η would he 0 in a next step if th controller input f _t *. a MM

Equation 10

| ⁽ ½8j where the next controller input, i .k- i is e ual ^" to the previous controller i put ¾ plus the product of the inverse feedforward transfer function Gj ' and the previous error ej _f . Note that the desired output and therefore the previous error ¾· may not be known, to the system. However, the previous error ¾ may he inferred by inverting the user response function G:-. yielding:

.Equation 11

ft*69) where the previous error can be written as th product of the inverted user response function G$.f ^} and the user iapnt Suppl ing Equation 1 V$ .formulation, ofet into Equation 1.0, it can be seen that the next controller input to achieve ^' 0 tracking error should be given as:

Equation 12

$70j where the next controller output is equal to the previous, controller output plus the product of the inverse feedforward transfer Junction Gjj ^"! and. the inverse of the user response function. GH ^'s and the human output u¾. Recalling the relation between the feedback and the feedforward transfer functions in Equation -6,. equation 12 can be rewritten, as:

Equation 13

[071 where the nest controller input Vk&i should be the previous controller input ¾k plus the product of the in verse feedback transfer function 0%Γ* and the ^' previous use input i¾. Note that the exact feedback transfer ilmetion _¾ may not be available- to die controller, -accordingly _* a model of th feedback function may be used, G¾. Usin the mode! of die feedback function, the next controller input rn¾y be written as:

Equation 1

where the next controller input equals the pre vi CMS controller input % _k plus the product of an iteration gain, p, the inverse model of the feedback function ' and the previous user input u&. The iteration gain p may generally be selected such that the iterative scheme converges, and: an initial ou put ¾o may be considered to be 0. The expression in .Equation 14 may be used to impie.me.nt the controller (e.g. the computing system 104 of Figure 1 and/or machine controller 310 of Figure 3 and/or block 204 of Figure 21 since the user input. % is available to the controller as well as the mode! of the feedback function.

[073J For -analysis purposes, note that Equation. 1.4 may be re-written using Equation

Equation 15

(07 I where ¹ is the inverse of the feedforward model, G¾ that is related to the known, models G _¾ ,& of the user response, G _¾ . a d system., G, respectively, such that: C½ Gii(m)G _f f{m)

Equations 16 and 17

[675 j So the stored feedback model (e.g. stored In mod l(s) of feedback transfer

.function. 1.16 of Figure .1 ) may be a representation of ύ&, C ^{' 1} , dn, or other mathematical combinations or relations thereof.

[W76] in some examples, an analytical-verbal model (e.g. a crossover-pilot model) may be used for {¾, which .may describe human motor response for manu l control tasks. The analytical-verbal model for Gu .may be expressed as;

Equation 18

7j where K _P is the operator static gain, ¾ is the effective time delay, and ·¾ is the equalization, time, constant. This generally refers to the analytical portion of the mode!, while the verbal portion ma refer to empirically validated adjustment rules for different task conditions. A«aly1ical«ver¼l models have been used with second-order svstems G of the form

Equation 1

[678] Analytical- erbal models are described, tor example, in. D. T. McRiier and E. S. rendel, " amematkal models of human pilot behavior," Tech. Rep., 1974, which publication is hereby incorporated by reference in its entirety i r any purpose,

|f>79) Figure 4 is a schematic illustration of a system for providing input to controlled equipment arranged hi accordance with examples described herein. The system includes combiner 402, user response 404, user feedback response 40 ^' 6, user machine interface 408, machine controller 410, combiner 412, controller model 416, and

■controlled .system 418, The desired output signal is represented in Figure 4 as y ₍ j. That desired mput. signal (e.g. desired trajectory) may be, known only to a user of the system, which ma fee a human user. An actual output of the controlled system 41 8 which may be implemented, for example, by any controlled e uipme t described herein, such as the controlled equipment 112 of Figure 1 } is represented as . There is an error, shown as e in Figure.4, between the desired output y,i and the actual output y. The error is provided by the combiner 402 whic subtracts the actual output y from the desired output y to yield the error, e. For example, y« _$ - ~ e. The error may prompt a user response 404, shown as Cm in Figure 4. User r sponse 404 .may include two components - a user feedback response 406, given as ¼ in Figure 4, and a user machine interface 408 given as _s in Figure 4, The user feedback response 406 may be implemented using measured frequency response data in some examples. Fo example, during a training process, a . user m provide inputs responsive ^' to a variety of errors e, and the measured frequenc response data may be used to provide In some examples, nominal parameter values may be used for the ^' human feedback response model, j¾. The nominal parameter values may come from prior observed results or from, knowledge of how the human reaction takes place.

[680j Generally., the user feedback response 406 may refer to the action a user ma take responsive to the input e. The action ma be to move a portion of the controlled system 418 and/or to take an action in or on a user interface to the controlled system 418 and/or th ^' .apparent controlled system 414. The user ^' maehrae interface 408 Kj may generally refer to tire correlation between the user's action and a signal or other direct input to the controlled system 418 and/or apparent controlled system 414, In this manner, the user's action may result in a provided response u _¾ in light of the error e. The model between and the: user response i¾ is .the human response Gg.

[081 Examples described herein may provide a machine controller 410 which may calculate an adjustment to the response the adjustment may foe shown in Figure 4 as u _e . The machine controller is shown in. Figure 4. as implementing a function G _c between the user response ¾ and the adjustment ¾. The machine controller 41.0 may be implemented, for example using the computing system 104 of Figure 1. The combiner 412 may combine the adjustment u _c and the user response such as by adding the responses,, to provide an adjusted input u for the apparent controlled system 414, Given the input u, the apparent controlled system 414 which may be modeled as the function GA shown in figure 4, may provide an output y. The output y may be ted back and compared at combiner 402 with the desired trajectory resulting in error e, and the process continues.

} Generally, the combiner 402, user response 404, user feedback, response 406, user machine interface 408, machine controller 41.0,· and combiner 412 may he implemented using and/or be analogous to the combiner 302, user response 304, user feedback response 306, use machine interface 308, and machine controller 310 of Figure 3. In examples such as that shown in Figure ^" 4 _S the controlled system 4.18 may differ from the controlled system 314 of Figure 3 in that the controlled system 418 may not have an existing user-response odel (e.g. a model stored in the model(s) of feedback transfer function. 1,1.6 of Figure I ). For example, the controlled system 41 8 may ^' be- a higher-order or more complex system.

} Accordingly, the controlled system 418 ^' ma be provided with a controller mode! 416, modeled as the function Gy in Figure 4. In this manner, the apparent controlled system 414 includes both the model G,- and the controlled system 418, G. The controller model. 41, 6 is selected such that the combined a arent controlled, s stem 414 has a behavior for which a user-response model is available (e.g. is stored in the model(s) of .feedback transfer function .1 16 and/or has been observed or described), hi this manner, the methods and systems described herein from improving output trajectory of the controlled system 418 may be used.

- The controller model Is selected such that

Equation 2.0

} where Q _A is the. apparent system, 6 _r is the controller model, and G is the controlled system model. The controller model G _r is selected to be a modeled inverse of the controlled system model G. G, _\ may be used in. place of G in the previous equations •and ^' description.. provided with reference to Figure 3 and a .known user-response model may be used to improve trajectory of the apparent controlled system 414. [886) Selecting the inverse controller model, G _(t equal to a modeled inverse of the controlled system mode! G _s may be performed in a variety of ways,

[087| in example ap lications, complete information of the controlled system, G may not be available, if no (or limited) prior knowledge of the controlled system, O is available, incremental learning techniques such as adaptive inverse control may he employed to obtain an appropriate controller Gr, In som examples, a model, G of the controlled system, G may be available, in which case feedfonvard inversion techniques may be used.

[088} ^' Figure 5 is a block diagram of online inverse computation system for an example system G. The computing system 104 of Figure I may, in some examples, perform the described online inverse computation. Figure 5 depicts an online .feedforward ^' inversion technique used with a known -model, G represented in state space form as, :( 1 ** {* ) - - Mm ($ } (i ) €x(t )

Equation 21

{089] The relative degree r of the system G may imply that the input u appears the τ ^Λ derivative of the output y as.

I where B _v is invertihie. The inverse input ΰ(ΐ) for tracking of an output trajectory y S'd may then be given as:

Μ(β )™ f^|i -r ii fr )—■ ( ··· . .. ' AVi/

Equation 23

[891) where e(f) ~ (ej(t) ... ¾-{ί}) is art error vector with e- f ? ™ ( v( )— ) > fer ™. j , 2. > , , > r.

Equation 2 2 ! j§92) and the feedback gains K ~ k; .... k _* j are chosen such that the poles of the error dynamics are on the left half of the complex plane. Note that ia the current setting

|tl93| Using examples of this inversion process in the control loop, a model of the feedback transfer function G¾ may be found using GA for G in Equations 2 and 3 mid using Equation 20, yielding:

Equation 25

|fl94| where G» ^::: {ft¾ and 1% can be found usin the equation for Gji from Equation 18. This may be used in the iterative inversion-based techniques described herein to p ovide the controller input is to achieve output tracking by the controller.

\ 9S\ in some examples, preview-based, inversion may be used, In some ^' examples, H _¾ may be written as ¾J 1

Equation 26

wher _p is the user static gain. τ _£ is the effective time delay, tj, is the lead time constant, and x\ is the lag time constant. Equation 26 may be provided in examples where, the time delay ¾ of the apparent system perceived by the user 3 _A is not too large, ^' The human-feedback response iodel in. Equation 26 may he used for computing estimate of h uman intent in methods described herein.

\%97\ Examples of preview-based inversion of a controlled s stem model G are now described. Preview-based inversion may be used to generate a.o apparent system from a controlled system (where, for example, the controlled, system may be more complex than for which .a user-feedback model is available). Figure 6 is a schematic illustration of an online inversion of a controlled system using a. preview-based online inversion technique. The preview-based online inversion techniques described, herein may be implemented using computing systems described herein, such as the computing system ί 04 of Figure 1.

Referring back to Equation 22, the iayerse system may then be written as

Equation 27

hetx τ _η Α - (&ς )} τ ^~

^' Equations 28

[0Ί 00} with T representing a coordinate transformation,

Equations 29

[010:1 S and Τ is chosen, such that T is invertible.

|©i02j Note that the .eigenvalues of A„ _lv re the zer of the original system in Equation 21. Accordingly, for a system with hyperbolic zero dynamics (e.g. none of the zeros of the system in Equation 21 .lie on the imaginary axis, of the complex plane), a. •transformation U ma be .found such that the i ernal dynamics in Equation; 2? can be decoupled, into a stable subsystem ( ¾) and an unstable- subsystem ( ¾):

Equation 30

0103) where

Equations.31

The individual subsystems in Equations 31 may hav bounded solutions given bv

Equations 32

«05 The reference internal state trajectory s -f may then be obtained

^■ Equation 33

[0106] and the inverse input, can be found using Equation 27 as

Equation 3

[0107] Complete future information of the desired output and its derivatives, ¥ _< j maybe needed to obtai the bounded solution < to the unstable subsystem, as seen in Equations 32. Thus, the desired output may need to be specified a priori, which may restrict the us of inversion t off-line scenarios (e.g. training-first). A preview -based inversion technique may be used to. overcome this limitation, by computing as approximated solution to the unstable subsystem σ» by using a finite amount of preview information, of the desired output and its trajectories, ¾ which may be given by:

Equation 35

0.lfl8] where ^ is the post-view time and T _Wt . is the preview time during which ¥ _ff (. is known. A suitable numerical integration scheme, such as zero-order hold (ZOH) integration may be used with sampling time T _s for on-line implementation-such that the integration in Equations 35 may he evaluated a

Equation 36

[0109] where ¾ ίί,ί) - Υ^ΐ-ΗΤ _* } and ni ^~ T _{W f _s and n "Ίρ, ί, are both assumed to be even integers. The weighting matrices for ike above numerical integration, scheme may be gives by

|¾il - e , ] - (i ^{: ' :} / ) Bv>

Equation 37

&110I The error in computing the inverse input ¼ _BV and therefore the tracking error e _y (t) may be made arbitrarily small by choosin large enough preview time T _pre and post-view tifiie and a sufficiently small sampling isrne T

j0 J Ϊ 1| In sorne examples, the input u(t) may be noisy. Accordingly, the input may be filtered in some examples. Figure 7 is a schematic illustration of a cascade of two first- order Iowpass filters which may be used to find a filtered signal and its time derivatives. The iowpass filters may be implemented., for example in hardware, software, or combinations thereof, and may be implemented by the computing system 104 of Figure 1.

(0II2| The cascade of first-oakr Iowpass filters m Figure 7 may he used to obtain filtered- input Ci, given by

4(f) :::: ; (Ui{l i,

Equations 38 j§H3j The combined signals including the first and second time derivatives (indicated with one and two .dots, receptivel ) may b -given, y

fM 141 such that the r 2 time derivatives Y ₄ of the ihput

Bqttation -40

{01 l.S j may be fed into the inversion scheme shown in Figure -6 to obtain the inverse input i½v.

{0-U6| Accordingly,- using examples of the finite preview information inversion, a model of the- feedback transfer .function Ci¾.may be written as

C¾¾ €¾5 ¾ 1©}

Equation 41

{0IJ7j where (assuming exact ^' inversion thai cancels the known system dynamics, G except for the potential previe delay, T _j»B! , which may not occur in all examples) the model of the apparent system, GA may be given by

Equation 42

[Qt Wl with itte time delay % equal to the - preview time- f _pfS! (¾™ T _re ), Additionally, fee model of the liser-teedback response GH in Equation 41 is Cm ~ K(ft& and H¾ is provided in Equation 26. The resulting feedback model G _¾ fro Equation 41 may be used to itemtively find the controller input u that achieves tracking of the intended output,

|ί>1 ϊ9] For example, _, at ^' each iteration step k an input may be found as Equation 43

i 20| where pirn) Is the frequency dependent iteration gain.

EXAMPLE MPLE EN ATIONS

Wi l] An iterative eo-iear.nitig algorithm was evaluated in an output trajectory tracking experiment wit a human in the loop. The subjects (n 9), 8 male and 1 female, were volunteers with ^' .age rangin from 20-50 years, The system output, y arid the .reference trajectory. ¾ were observed by. the human through a graphical user- interface (GUI). A circular cross-hair followed the system output y while a red cross followed the reference trajectory, y& The human user, estimated the error, e ^K: y<j - y in the output, and provided the input, ¾ through, a human-machine interface (ΗΜΪ), in this case a Microsoft Kineet camera, by moving his/her right hand from side to side. The■.trajectory of the. right- hand is then tracked: using the Skeleton trackin routine provided in the Kineci Software Development Kit (SDK), The BMi then computes the difference between, the tracked ^coordinate of the ri ght hand from the origin of the Kineci frame of reference, which is then multiplied b a gain, K : to obtain the human input, uj,. The gain K.J is chosen such that, maximum range of human input is limited to arm motions without the need for large whole-body movements by the human user. The human input, u¾ is added to the controller input, _* 1» obtain the total input, ¾ ^« _¾ *· u that is applied to the system, G. The system is simulated in a computer to enable potential experimentation with different types of systems, but, coul be replaced with a physical system (e.g. controlled equipment described herein). Mote that in this setup, th human has access to the desired output, y _< j nd the system output y, while the controller G _c does not have access to ,t, as specified in Constraint 1. Also, the controller, <¾ has access to the model of the closed-loop system, ¾·„ to meet Constraint 2.

j til 22] The analytical-verbal model described herein is used to capture the human motor response for manual control tasks, A second-order system, G is used-— the parameters used in the experiments are defined in Table 1. TABLE 1

Parameter Values

]

The resonance frequency i« _> where « _B ^r?: 2s , of he critically damped system G was chosen to be below the smo th-piirsuil ^■ visual tracking limit of f ^::: O S Hz for typical healthy h man users. It is noted that such linear time-invaria second order systems, G have beet) used extensively in literature to model human machine interactions, e.g., for vehicle: steering angle control (such as described in S. Fa anrs, H, Zamzori, S. A. Maxlan, and M. Zafcari , "Modeling and simulation of vehicle steer by wire system," is .Humanities, Science and .Engineering Research (SHUSER), 2012 IEEE Symposium on. IEEE, 2012, pp. 765-770), and for modeling human arm movements when constrained by active robotic systems (such as described in J. M. Dolan, M, 8. Friedman* and M, L. Magurka, "Dynamic and loaded impedance components in the maintenance of human arm posture," Systems, Man and Cybernetics, ΪΕΒΕ Transactions on, vol. 23, no, 3, pp. 698-709, 1993, and . Hollerbaeh and H. Kazeroom, "Modelin human ar movements constrained by robotic systems," in Proceedings of the Winter Annual Meeting of the American Society of Mechanical Engineering (Advances in Robotics), 1992, pp. 1 -24). The. three aforementioned papers are hereby incorporated by reference in their entirety for any purpose.

j{>!24| The desired trajectory, \¾ tor this visiio-manual tracking task was chosen to be representative of a reach and retract movement. The acceleration profile of the desired •trajectory is defined as.

ate as given in Table 1 , and t _f<S i™ 0.5 s.. To allow visual display of the desired output and actual output to the human user, the target frequency, l of the desired trajectory, y& was chosen to be below the smooth-pursuit visual tracking limit of *" 0,5 Hz for typical healthy human users, e.g. fx < f ^* r " ^: 0:5Hz. \M27\ The frequency response of the feedback loop, is obtained from moM- estimatiori trials. .During the model estimation, the desired output, ya is specified to be pure sinu oidal trajectories of die form v t) = Λ $m(2n ft). for r > 0 j. 129] i h the iterative controller deactivated (sBanuai control. task)- By performing. a sweep through the frequencies of interest (by varying the desired-output frequency, f }, ■fee human response G» is computed at each frequency as

G _# ( #) ) = — ,.... .Λ....... _; ............ _{Γ λ} ω 2π ί

[«S38j · ^■· >

jei31j Ten trials are performed, at each frequency, f in the range [tU ; i ,0] Hz. The resulting frequency respou.se is used tp .St the parameters of the cross-over model. First, the exponential phase component from the human, response G _¾ is removed

¾ (Ci) | - ¾ ( ) ^¾ ® , :::::

\MM\

[8133] where different values of the time-delay parameter,. Ta e [0,1] s are chosen.

These: constitute, typical time-delay values found in literature. Next, the best-fit . transfer function, d« ^* the delay-removed response,€¾ in th above equation is computed using the Mail

[8134]

[0135[ to obtain the time-delay dependent values of the parameters, _p , τ¾, τ _& The optimal values for the parameters _r <; ·¾, and the time-delay, % are then obtained by minimizing over the time delay ¾

[81371 Where || , !] _¾ is the standard 2-norm of a. vector.

[8138] It is seen that the linear crossover model, i¾ is a good .representatio of the human response for low frequencies, but larger deviation is seen at high frequencies in, 3 ! which compares the fitted model, £¾¾ and the frequency response (fate, <¾ for Subject 2. Such deviation at frequencies hig er than the smooth pursuit limit of 0.5 Hz is expected because visual, observation of the error Is expected to involve large nonlinear remnants at .high frequencies, which may sot be captured by the linear model.

[0139] The fitted .human mode!, 6¾ is thai used, in conjunction with the known mode! of the controlled plant, G;

[M l] to obtain the feedback-loop model, G¾ as in Eq, (½). The estimated parameter values in the above model ar as given in Table 1. Note that 10% errors are present between the parameters of the model 6 and actual system. G.

| ^' Θ1 2| A total of k ^rr: 21. iterations were performed, divided into three distinct control regimes: (i) hurnan feedback only, without machine learning. ¾ , for iterations k e

[1,10], (u). human-machine shared control, using human-guided inversion-based iterative control, for iterations k e [1 1 ,20] with zero initial controller input, , ¾ ₍ y ^::: G, fiii) machine open-loop control (k ^:iK 21.) with learned control input,. u& ^::: ¾,¾f without human feedback, ¾,:?¾ ^« 0. A rest period was provided at the end of every ^' iteration to enable resetting of the initial conditions.

fti J43j Figures 8A.-8D illustrate convergence of inversion-based iterative learning control for the target frequency, ft ^: ~ 0.125 Hz for Subject 2. FIG. 8A illustrates system output, y*. during; human-only control (fc e [ 1 ,1.0]), FIG, SB illustrates system output, _ during shared human-machine iterative learning control {k ε [ 1 1 ,20]). FIG. 8C illustrates tracking error, e½ =" ¾ - y«) at the end of human-only control (k -~ 10). FIG. 8D illustrates tracking error, at. the end of shared control (k ^; -~ 21). The output tracking, performance is improved by the iterative learning controller when compared to the human-alone perionoanee, as shown in Figures; 8A-D for the target frequency, ft 0, 125 Uz for Subject 2, whose performance is close to the average performance level of the nine test subjects. Note that the tracking error e?i at the end of shared human-machine iterative learning Is lower than the tracking error ut in. Figures 8A-D with the hunian-onlv control \M44\ Nominal parameter values from existing literature for the analytical-verbal human, feedback model, ¾ can be used instead of erforming model estimation trials to obtain these parameters from the experimental frequenc response data, (¾· For ■example, the time-delay parameter, ^" ¾ has typical alues in the range 0.1-0.2 s for 'healthy individuals, while the equalisation time-constant, T. _J , depends on. the crossover frequency, ω _¾ of .the open-loo .response, |0«(<£)0(ω}| as, 1/ ¾ «ω* where o¾ s¾ 5 rad/s for second orde systems with high natural ^' frequency, > 1/ ^· ·τ¾. Table 1 lists the nominal values of these parameters, ¾ and The overall gain, K _p may be characteristic .Of the user preferences such as range of and motion, and can be determined .from the initial set of th model estimation trials without the need for additional trials to obtain the f equency response data for deterau ng the human feedback response model ¾?.

$145] From the foregoing it will be appreciated that, although specific embodiments of the invention have been described herein for purposes of illustration, various modifications may be made without deviating from the spirit and scope of the invention..