CONTROL OF KINESTHETIC HAPTIC INTERFACES IN VR APPLICATIONS

Cartesian control algorithms are presented for 6-dof force-reflecting hand-controllers (FRHCs) used for simultaneous operator position/orientation (or rate) commands to a virtual reality (VR) system and virtual force/moment kinesthetic reflection to the operator. The commands and kinesthetic feedback are transferred in Cartesian space. The task force/moment (wrench) dominates while features are provided to reduce operator loading: virtual payload and FRHC gravity compensation, input channels to easily separate 6-dof inputs with one hand, constant-force return-to-center, and FRHC damping to improve relative stability. In experimental implementation, the “VR system” was a real remotely located teleoperated robotic system with real sensed task wrenches. Experimental results show that the algorithms are effective for reduced contact wrenches and increased telepresence quality in practical tasks. The methods in this paper are suitable for kinesthetic haptic display in virtual environments.


INTRODUCTION
Force/moment (wrench) reflection to a human user can greatly improve telepresence via kinesthetic haptic display in virtual worlds.The "conventional" visual and auditory VR feedback is greatly enhanced by allowing the user to feel weight, inertia, friction, stiffness, and contact of virtual objects in six degrees-of-freedom (dof).VR systems with kinesthetic haptic feedback have great potential in virtual design, manufacturing, rapid prototyping, virtual training, surgical training, remote surgery, education, and entertainment.Tactile haptic feedback, also having great potential in VR systems, is not considered in the current paper (though some tactile sensations can be generated using kinesthetic feedback).For an excellent overview of tactile and kinesthetic haptic interfaces in VR, please see Burdea (1996, including 21 pages of references).For a general overview of VR systems, see Pimentel and Teixeira (1995).
Many authors have presented results in force reflection systems.Dick et.al. (1994) present experimental results for a kinematically-similar Kraft master/slave system controlled joint-to-joint.They find that force reflection to the operator reduces slave joint torques.Kazerooni and Her (1994) consider the dynamics and stability in control of a force-reflecting haptic interface.Repperger et.al. (1993) present haptic feedback in handicapped-assist applications.They include the operator's dynamic response to determine if force or position control is preferable for minimum-time tasks.Hannaford and Kim (1989) study the effect of time-delay teleoperation in kinesthetic force reflection systems.Operator performance based on task time and applied forces degraded linearly with increases in time delay.This paper presents a general controller for effective wrench reflection with 6-dof active hand controllers, in Cartesian space.Such force-reflecting hand controllers can be used for input and output.
As input, the pose (position and orientation) of the hand-grip guided by the operator can be interpreted as pose or rate commands to be sent to objects in the virtual environment.As output, the remotely-sensed task wrench or computer-generated virtual task wrench can be sent to the active hand controller so the user feels the task wrench.Both input commands and output wrenches are transferred in Cartesian space.
Control algorithms are developed which simultaneously provide Cartesian wrench feedback to the operator's hand and provide desirable characteristics to reduce operator loading and fatigue.These characteristics include hand controller gravity compensation, constant-force return-to-center (for rate inputs), virtual channels so the operator may easily separate the 6-dof inputs with one hand, deadbands, and hand controller damping to improve stability.The techniques of this paper have been implemented at NASA Langley Research Center in practical control of FRHCs interfaced with experimental telerobotic manipulators.The FRHCs were used to command both virtual robot system models and remote hardware.

SYSTEM DESCRIPTION
For presentation in this paper, it is assumed that the force-reflecting hand controller (FRHC) is used by a human operator to command objects in virtual environments.Computer-generated virtual wrenches are sent to the operator via the FRHC.
It is assumed that the FRHC is a jointed device having six axes which enable six independent inputs.
It is further assumed that each axis has an active motor; by coordinating the six motors Cartesian wrenches may be exerted on the operator's hand.Virtual objects in 3D virtual environments can be commanded in six-dof (three translations and three rotations).The object commands and wrench feedback are transferred in Cartesian space, which is natural for humans.Figure 1 shows two FRHCs which have been controlled with methods of the current paper: the Kraft Master (Kraft, 1989 andWilliams, 1991) and JPL (McAffee and Ohm, 1988) FRHCs.

Figure 1. Kraft Master (left) and JPL (right) FRHCs
Figure 2 shows the command and feedback flow for this paper.The operator moves the FRHC to a series of positions and orientations (poses).At each control cycle, the current FRHC Cartesian pose is interpreted as a rate or pose command to send to the virtual object or objects.The modeled static and inertial loads due to virtual object mass, mass-moment of inertia, and dynamics can be included.The moving reference frame {MRF} is the frame under Cartesian control, rigidly attached to the virtual object of interest.When the operator has achieved contact with the virtual environment, the task wrench is sent to the control computer which in turn commands the FRHC so the operator feels the contact wrench.Virtual contact may be simulated with 6-dof mass/spring/damper models.

VR
where any valid rotation convention may be used (e.g.Euler Z-Y-X α β γ , , ).The same pose (position and orientation) information is contained in homogeneous transformation matrix j i T , which gives the pose of frame {j} relative to frame {i} (Craig, 1989).A Cartesian rate vector is: and a Cartesian wrench is: where the wrench is applied at frame {j} and expressed in {i} coordinates.
The next section presents a general Cartesian controller framework to implement the generic system depicted in Fig. 2. In addition to transferring input manipulator commands and output task wrenches, features are included to decrease operator loading, facilitate separation of 6-dof commands in one hand, return-to-center for rate commands, and increase stability.

CARTESIAN CONTROL FOR FRHC
Figure 3 shows the control flow for the implementation of the FRHC commanding inputs and displaying wrenches with a virtual system in Cartesian space.The following subsections describe the control algorithms of Fig. 3. Figure 3 assumes Cartesian rate inputs; the difference for Cartesian pose inputs is minor (Section 3.5).

FRHC Cartesian Input Commands
Let us start with the block "FRHC Including Operator" in Fig. 3.The user must first define the desired FRHC reference pose G T 0 0 via a FRHC switch.G T 0 0 can be any convenient pose in the FRHC workspace which represents zero Cartesian input to the virtual environment.The switch in Fig. 3 indicates the operator may redefine this pose at any time.This feature is intended to decouple the Cartesian input from the FRHC base frame and allow generality for commands.
During any control cycle when the operator has moved the FRHC grip from the reference pose, the input command is determined as follows.First the FRHC joint sensors are read and FRHC forward kinematics calculates the current grip frame pose relative to the FRHC base frame (note G T 0 0 was also calculated using the form of Eq. 1): For both Cartesian rate and pose commands, the next step is to determine the "difference" in the current and reference grip poses.The translational difference is a simple algebraic subtraction: { } 0 0 0 0 0 However, the rotational difference is not as simple because there is no vector representation for orientation.A relative difference matrix is used, relating the orientation of G relative to G 0 : At this point, the Cartesian pose and rate input cases differ.For Cartesian pose input (not shown in Fig. 3), ∆X (scaled by gain matrix K P ) is commanded directly to the virtual object, relative to the reference object pose.In Fig. 3 all gain matrices K m are order 6x6 and (generally) diagonal matrices of gains.
For Cartesian rate input, the translational terms are simply the first three terms of ∆X scaled by the first three K V diagonal elements (units sec -1 ).Since we cannot use small angle assumptions, the rotational terms are calculated using the appropriate rotational rate kinematics transformation (Euler Z-Y-X α β γ , , , adapted from Kane et.al., 1983): Because a static FRHC pose must be converted into a Cartesian rate, we use γ γ = , β β = , and α α = in Eq. 5. Then the angular velocity from Eq. 5 must be scaled by the second three K V diagonal elements (again, units sec -1 ).The total rate command to the control frame {MRF} is X MRF , formed from these translational and angular rate terms.Note Eq. 5 is not shown in Fig. 3, but may be considered to be lumped in the K V block.The virtual simulation must respond in real-time to the operator's Cartesian commands.

Cartesian Task Wrench-Reflection to FRHC
With either Cartesian pose or rate inputs ∆X or X MRF from the FRHC as above, we have now proceeded to the Fig. 3 block labeled "Virtual System with Haptic Feedback".The virtual object performs the commanded motion while a haptic feedback algorithm calculates the feedback wrench S F at each control cycle.To reduce operator loading, the next block on the lower return path in Fig. 3 removes the virtual payload weight and moment terms from S F .This block is bypassed if the user wants to feel the simulated mass properties.In the same block, a dynamics model may be used to include inertial loading, if desired.
The task wrench (when contact is made between the virtual human hand / object / environment) is generated in a frame {S} but we wish to give the operator the task wrench from the virtual object control frame {MRF}.(In the telerobotics application, {S} is the force/torque sensor frame while {MRF} is the frame under control.)Therefore, a rigid-body force/moment transformation is required determining the equivalent {MRF} wrench based on the simulated {S} wrench (Craig, 1989): Next gain matrix K T is applied to scale the task wrench based on FRHC capacity and operator safety to yield the wrench G T F to be felt by the operator in the FRHC grip frame {G}.Since the FRHC Jacobian matrix is generally derived to relate the last FRHC frame {W} (for wrist) to the FRHC base frame, another rigid-body force/moment transformation is required on the FRHC side.This has the same form as Eq. 6, substituting G S → and W MRF → . Note this applies to FHRCs with spherical wrist design.For non-spherical wrists, this last transformation is unnecessary, if the Jacobian is derived for {G}.Also, some FRHCs (such as the JPL FRHC) have the {G} and {W} identical which also obviates the need for this last transformation.
To calculate the required six FRHC joint torques (and forces if there are prismatic joints), the Cartesian wrench to joint torque statics transformation involves the Jacobian matrix transpose (Craig, 1989): where the FRHC Jacobian matrix J is a function of the FRHC joint values.Note to use Eq. 7 directly, the Jacobian (relating {W} to the base frame) must also be expressed in {W} coordinates.Otherwise W T F must first be transformed to the coordinates of J via coordinate rotations.
These two sections are sufficient to command a objects in a virtual world and feed back task wrenches simultaneously with a FRHC in Cartesian space.However, the next section presents additional features of Fig. 3 to improve operator loading and FRHC stability.

Features for Improved Operator Loading and Stability
One benefit of wrench-reflection to the operator is increased feeling of telepresence and realism in the virtual environment.However, one drawback is the potential for increased operator loading, including fatigue from resisting wrenches through the FRHC and supporting a portion of the FRHC weight.The operator can choose whether or not to include the virtual object weights and moments, as discussed in the previous section.
In the current section, FRHC gravity compensation is presented to further unload the operator's arm.
Also, for Cartesian rate inputs, a unique return-to-center method is developed which assists the operator in finding the zero input FRHC pose when zero inputs are desired in between commanded motions.For both rate and pose inputs, a damping term is also added to improve relative FRHC stability.All three of these features are added as a 6-dof wrench to be applied at the FRHC grip.Therefore, it is crucial that these operator aids do not mask the task Cartesian contact wrench.This will be discussed later.
FRHC Gravity Compensation.Some FRHCs are gravity balanced by design (such as the JPL FRHC).
However, many others are mini-articulated robots which must be supported by the operator.FRHC gravity compensation can apply a configuration-varying joint torque vector so the FRHC supports most (theoretically, all) of its static weight.At the CG of FRHC link i, the weight m i g acts.If a fictitious force f icomp = m i g is provided equal and opposite of the weight vector, that link will be balanced.This f icomp may be transformed to an equivalent wrench at the FRHC grip using the appropriate rigid-body force/moment transformation of the form of Eq. 6, with appropriate indices substituted.The sum of balance wrenches for all FRHC links will yield the single FRHC grip wrench ( G GC F in Fig. 3) required to unload the operator's arm by commanding the FRHC to support its own weight.
Constant-Force Return-to-Center and Walls.For Cartesian rate input commands, virtual objects will move with constant velocity when the FRHC Cartesian pose is different from the FRHC reference pose.
Therefore, a return-to-center (RTC) force should be provided to assist the operator's hand in finding the zero input FRHC pose.As a first attempt, this RTC force was calculated using Hooke's law with a virtual spring: G R R F K X = − ∆ .The FRHC grip wrench is calculated for each Cartesian axis (3 translations, 3 rotations) independently; the negative sign is to draw the operator's hand back toward the zero pose.However, it was found that the FRHC workspace far from the defined reference pose generated large RTC forces unnecessarily due to the linearly increasing relationship.
Therefore, a novel constant-force return-to-center (CFRTC) approach was developed.Figure 4 shows the CFRTC force as a function of scalar displacement ∆X i from the zero reference, for one of the 6 Cartesian axes.∆X i represents any one of the six terms in the relative Cartesian pose ∆X .

∆X i
i , , , , =1 2 6 are found from FRHC joint sensors and Eqs.1-4.The magnitudes in Fig. 4 are arbitrary and must be determined for specific FRHCs based on performance requirements and FRHC workspace.The CFTRC is symmetric about ∆X i = 0 ; each side displays three distinct (but continuous) regions.The first is the deadband, serving two purposes: a) providing a small region of zero input surrounding the zero pose; and b) providing a parabolic wall which the operator must overcome if an input is to be commanded in that particular Cartesian axis.The second, largest, zone is the working range which provides the CFRTC (as opposed to Hooke's law) virtual spring.The third zone provides a stiff virtual spring to alert the operator when the edge of the FRHC workspace is encountered.In practice it was found that this stiff spring was unnecessary so the flat CFRTC zone was extended to the workspace boundary.In this case, the operator must be aware of the workspace boundaries, but the effective FRHC range is extended.

Figure 4. Constant Force Return-to-Center for One Cartesian Axis
The i th term for the Cartesian CFRTC wrench G R F is expressed in Eq. 8 (ignoring stiff virtual spring of the third zone).Note translational pose terms correspond to return forces while rotational pose terms correspond to return moments.

( )
where the constant ∆X iDB is the i th axis deadband value and a i is the i th axis parabolic constant.If > , ∆X iDB must first be subtracted from ∆X i before it is used in a Cartesian pose or rate command.Figure 4 and Eq. 8 are represented on Fig. 3 by the virtual spring characteristics K R (more complicated than the other Fig. 3 matrix gains due to the different zones).
Damping Term.To increase relative FRHC stability, a damping term is added.If the FRHC pose is static, there is zero damping term.However, if the operator is making FRHC pose changes with respect to time, the damping term applies a resistive wrench G D F (opposite to the velocity direction of each Cartesian pose term) at the FRHC grip.This serves to dampen rapid changes in the manipulator Cartesian commands.
The ( ) ( ) ( ) The stability issue is important in the wrench-reflecting system of Fig. 2. If the operator makes contact between a virtual object and a simulated stiff environment at a high rate, a large wrench will be reflected, which pulls the operator's hand back.In turn, the virtual object will reverse, only to be returned with the command from the operator's hand recovering forward.This situation can lead to an oscillating instability.This is difficult to model due to a lack of a good model and variability for human operators.In the experimental system, stability was aided by the FRHC damping term, but operator training and heuristic gain tuning also helped stabilize the system.A future goal is to better ensure stability.
Total Assist Terms.The gravity compensation, return-to-center, and damping operator assist terms are summed to determine the required assist wrench G RG F at the FRHC grip: As with the task wrench case, G RG F must be converted to the equivalent wrench for the {W} frame (Eq. 6 with proper indices) before using Eq.7 to calculate the joint torques/forces to achieve the assist features.

Total FRHC Joint Commands
The total joint torques/forces commanded to the FRHC joints is the sum of those required for the task wrench and those required for the assist wrench.In order to ensure that the task wrench dominates, the assist wrench τ RG is first scaled uniformly to a set fraction of the FRHC capacity to yield τ RGsat .
Notice that a similar saturation could be performed so that the task wrench is allotted the remaining (majority) FRHC capacity.However, this was not done because it was found that when contact with the environment is simulated, the assist features became less important and the task wrench should dominate strongly.The total joint torque/force command is the sum of the task term and the saturated assist term: In order to calculate the final joint commands τ C , a final uniform scaling (saturation) must be performed.Also, gear ratios and torque calibration curves must be implemented.Now the discussion of Fig. 3 is complete.The operator feels any task plus assist wrenches and continuously updates the Cartesian commands to the virtual world.

Pose vs. Rate Cartesian Commands
The experimental systems implemented allowed both pose and rate inputs.Figure 3 is developed for the rate case, but the pose case is very similar.For the pose case, the velocity input scaling gain matrix K V is replaced by the pose gain matrix K P .The rotational kinematic transformation Eq. 5 is no longer required.The pose command is determined directly from the commanded ∆X applied to the virtual object reference pose.In pose mode, the CFRTC assist wrench is not required because a static FRHC pose yields a static pose (rather than moving with constant velocity).The deadband and virtual walls were first implemented but not found to be as useful as in the rate case.However, the same FRHC gravity compensation and damping terms apply well to the pose case.

IMPLEMENTATION ISSUES AND RESULTS
The algorithms described in the previous section were implemented for the Kraft Master and the JPL FRHC (Figs. 1) teleoperating kinematically-dissimilar 6-dof PUMA and 8-dof AAI slave robots in Cartesian space.This section briefly discusses implementation and results.
Three modes were implemented: 1) teleoperation of a virtual SILMA model of the slave robots with simulated contact wrenches; 2) teleoperation of remote slave robot hardware with real force/torque sensor feedback for the contact wrenches; and 3) simultaneous teleoperation of the virtual model and remote hardware using remote sensors to drive the virtual model and FRHC feedback.The focus was improved telepresence in the real remote system, so no virtual/real wrench sensor comparison was attempted.Our virtual models were limited to monitor graphics; VR immersion would greatly increase the telepresence quality when using kinesthetic haptic feedback.Figure 5a shows the virtual model of the AAI robot, while Fig. 5b is a photograph.

Switches
Each of the FRHCs had at least three switches.The most prominent on each was used as a deadman switch; that switch must be continuously held by the operator to send commands to the virtual/remote robot and receive wrench reflection back.A second switch was used to define the FRHC reference pose as shown in Fig. 3 and discussed in Section 3.1.This same switch may be used as an index button to command the entire slave robot workspace with a limited FRHC workspace, when in pose input mode.
The third switch was used to enable/disable wrench reflection from the task to the FRHC.The Kraft gravity compensation was robust enough to be enabled at all times (even without the deadman switch).
However, return-to-center, damping, and wrench reflection terms are not enabled without this third switch.The second and third switches do not need to be depressed continuously.

Torque Curves
The FRHC Cartesian control algorithms were implemented on the Kraft Master using manufacturersupplied torque curves.They were given as linear relationships between desired joint torque and counts to command to each active joint.For simple 1-dof Cartesian test task forces, the feeling at the FRHC was all wrong.The problem was traced to inadequate torque curves.Therefore, the proper torque/counts relationship was determined for each active joint, in both driving directions.A typical result is shown in Fig. 6.The dashed line is the linear relationship given by the manufacturer.The solid curve is a 5 th -order polynomial which represents a least-squares fit for the calibration data.When the 5 th -order polynomial calibrations were applied for each active joint, the simple 1-dof Cartesian test task forces had much higher fidelity on the FRHC.The joint torque calibration was found to be symmetric about the origin.Hysteresis is also an issue but did not significantly affect the operation of either FRHC.

NTRFC Results
In the same experimental system, a novel controller was developed where rate control is in effect during free manipulator motion, but this transitions naturally to force control when the manipulator comes into contact with the environment (Naturally-Transitioning Rate-to-Force Controller, NTRFC, Williams et.al., 1996).The enabling algorithm is force accommodation control where the manipulator automatically moves to alleviate any contact wrench.When the human operator continues to give rate commands after contact, an equilibrium condition is entered where displacement of the hand-controller is proportional to exerted wrench (no longer commanded rate).
This NTRFC can operate without a FRHC.However, if a FRHC is used in conjunction with the NTRFC, the force of the operator's hand on the FRHC is proportional to the task wrench applied by the slave manipulator.This mode was found to provide excellent telepresence and fine wrench control in contact.
An informal experiment was conducted to determine the effect of this FRHC teleoperating in NTRFC mode, compared to no slave accommodation or wrench feedback to the operator.Three trained operators were asked to complete a dual-peg-in-the-holes task multiple times.Half the tests used the FRHC/NTRFC while the other half force the operator to work "blind" in the wrench domain, without wrench reflection (and without the slave-side accommodation).For each case, the work exerted by the manipulator on the environment was calculated as follows (sum of all wrench component times Cartesian displacement component during the task): Table I shows the experimental results.It is clear that the FRHC/NTRFC yields lower work exerted to accomplish the task, for all three operators.The cases without force reflection or accommodation caused high wrenches (with more variation) to be exerted on the environment.This was intended for demonstration purposes only and not as a human factors study.

CONCLUSION
This paper has presented control algorithms for force-reflecting hand-controllers (FRHCs) used for simultaneous operator commands to a objects in a virtual environment and wrench-reflection back to the operator, both in Cartesian space.Pose or rate commands are allowed and the FRHC kinematics need not be similar to the virtual objects because the information is transferred in Cartesian space.The philosophy is to allow the task wrench to dominate while providing features to reduce operator loading: virtual object and FRHC gravity compensation, input channels to separate 6-dof inputs with one hand, constant-force return-to-center, and FRHC damping to improve relative stability.The methods were implemented on experimental FRHCs commanding virtual/remote slave manipulators.Experimental results demonstrate that FRHCs controlled in this manner are effective for reduced teleoperated task contact wrenches and telepresence quality.Kinesthetic haptic interfaces add an amazing dimension not present with VR systems using only visual and auditory feedback.

F
∆X vector is calculated via a simple difference in the current and previous ∆X values.In this case we have small angle motion (between control cycles) so the entire pose representation ∆X may be subtracted algebraically.G D is calculated by applying a diagonal matrix of damping gains K D (with negative signs) to ( ) ∆ ∆X .

Figure 6 .
Figure 6.Joint Torque to Counts Calibration

Table I .
Experimental Work Results