Touching the Void: Exploring Virtual Objects through a Vibrotactile Glove

(cid:178) This paper describes a simple low-cost approach to adding an element of haptic interaction within a virtual environment. Using off-the-shelf hardware and software we describe a simple setup that can be used to explore physically virtual objects in space. This setup comprises of a prototype glove with a number of vibrating actuators to provide the haptic


INTRODUCTION
Advances in information systems and technologies along with the declining cost of electronics have made it possible to make immersive virtual reality an affordable tool for entertainment, everyday applications and research. Low-cost stereoscopic 3D displays along consumer level full body tracking together with spatialized audio systems can provide highly immersive experiences in computer generated virtual environments.
The advances in vision and sound technologies though have superseded considerably the advances in haptic systems. This is largely because of the nature of the information conveyed, since vision and sound are by definition sensory modalities where the source of the sensory information can be placed at a distance from the receptor. The information in both modalities is propagated in waves, either electromagnetic or air pressure. The haptic modality though relies on direct collocation of the receptor with the source of the information. This creates the necessity for the source of the haptic information to have a physical aspect in contrast to the ethereal aspect of the audiovisual signals. Unlike audiovisual systems, haptic systems rely heavily on solving problems in mechanical design, actuators, real-time systems, user-object interaction modeling and human capabilities among other [1]. This physical aspect of the nature of haptics is what makes it so challenging and at the same time slow in evolution. This also is the aspect that will transform a virtual reality experience into a real life experience, by introducing physical consequences in actions within a virtual reality system.
Vibrotactile displays and systems have been developed in many configurations in the past for a variety of applications including navigation in virtual environments [2], improving motor learning by enhancing the auditory and visual feedback [3] as well as for sensory substitution [4]. Vibrotactile glove systems have also been developed independently for various studies involving virtual social interaction and rehabilitation [5] as well as telemanipulation tasks [6]. Commercial vibrotactile glove products [7] are available, although they are expensive and with a small number of actuators.
The system described in this paper demonstrates a simple and low cost solution for providing an element of haptic feedback from a virtual environment, by using a set of vibrators fitted on a glove, a microcontroller, a tracking device and a virtual reality client.
The design of this system is aimed at providing a starting point for the introduction of physicality within a low-cost virtual environment (VE) and creating a potential tool for exploration of the human haptic perception, through the vibrotactile modality.
As a proof of concept, a simple task was devised involving a virtual environment consisting of a set of two-dimensional hollow shapes. Within this VE, the users had to deduce the shape of the perceived objects relying solely on the haptic feedback obtained from the vibro-tactile glove by exploring the space in front of them. II.

MATERIALS AND METHODS
We developed a simple application using the XVR with a Kinect [9] and provides them with vibrotactile feedback by means of the haptic glove.
The virtual environment shows 4 objects and a simple model of a hand, as shown in Figure 1 supported by the XVR platform. Although the shapes are three-dimensional, we utilize only the two dimensions of the shapes (not taking into account their depths). The reason for this is due to the fact that we only obtain the positional data of the hand, without taking into account rotations. uses it to move a virtual hand model in the VR environment. We use ray-casting techniques to detect when the hand touches the virtual objects. We start by computing ray-to-object intersections for points around the position of every vibrator on the plane of the virtual hand. If the intersection was successful we store 1 as the collision value. We would store 0 otherwise. To calculate the intensity value that corresponds to a vibrator in the glove we average the collision values found around that vibrator during the ray-casting process.
The following subsections describe in detail the subsystems we developed to track a user, compute collisions with virtual objects and map those collisions to the haptic glove.

The Haptic Glove
The haptic glove assembly consists of a standard right-hand glove and 14 coin-type vibrators.
The glove chosen was an off-the-shelf glove used for cycling, wh The vibrators used are 10mm in diameter, coin-type vibrators with a rated speed of 9000 rpm.
For the construction of the glove assembly, the vibrators were initially mounted on small perfoboards, for ease of manipulation and connection soldering. Once on the boards, the assemblies were each glued at specific points on the glove, in order to provide stimulation on two points per finger and four points on the palm of the hand. The vibrator assemblies were placed with the vibrator side in direct contact with the glove material, to male DB-25 connector, along with the common ground of the vibrators on a separate pin, as shown in Figure 2.
A female DB-25 connector was connected via an Arduino prototyping board to the pulse-width modulation (PWM) pins of an Arduino MEGA 2560 board [10] (pins 2-13 and pins 45 and 46).
The microcontroller was programmed to receive 2 bytes to trigger the vibrators. The first byte received (from 0 to13) indicates the vibrator to be triggered and the second (from 0 to 255) the intensity of the vibration. The completed assembly can be seen in Figure 3.
The reason for using 14 vibrators in our configuration on the glove is mainly due to the physical size of the vibrators. Furthermore, due to the fact that not all areas on the glove are in direct contact with the skin (e.g. center of the palm), placing more vibrators in those areas would result in diffusion of the sensation and difficulty in stimulus discrimination. Fig3: The completed assembly of the glove connected to the Arduino board.

Tracking System
and input that data to our application. Our application obtains data from Kinect by means of the OpenNI framework.
Instead of using data provided by OpenNI directly in our application, we used the Hardware Accelerated Library for Character Animation HALCA [11] and a local library that maps the joint positions from Kinect to a virtual avatar. The use of these two libraries is not required at all to build the haptic functionality we present in this work and reading the positions directly from OpenNI to track the virtual hand model should produce the same results.
In our application, the data provided by OpenNI is read into the avatar mapping library and used to animate an avatar. Once we have the Kinect tracking mapped to an avatar, we read the position of the translation to the virtual hand model. Figure 4 shows the main elements involved in the tracking subsystem.

Haptic Glove Calibration and Mapping
We have implemented a calibration mode into our application that allows us to change the mapping between the virtual hand and the haptic glove. In the calibration screen we can move the virtual representation of the vibrators on the hand so that they match the position in the haptic glove. Figure 5 shows the virtual hand model with small cylinders that represent the virtual vibrators.

Collisions algorithm
We have implemented a ray-casting algorithm to compute collisions between the virtual hand model and the objects in the environment.
Our algorithm starts by taking the bounding box around the virtual hand and computes a grid of n by n cells. Casting n by n rays in a scene with m objects would have a time complexity O (m·n2). To optimize the quadratic behavior we cast rays for a small t by t cells kernel (t<<n) for every vibrator, thus resulting in an algorithm with linear time complexity O (m·v) with respect to the number of objects in the VE. The number of vibrators v is a constant in our system.
To compute the averaged collision value for every vibrator, the algorithm uses a t by t cells kernel to sample collision values around every cell that corresponds to a virtual vibrator in the previous mapping (Section 2.3). For each cell (i,j) in the kernel, view and passes through the center of the cell. For each ray, we perform a collision query over all the objects in the scene. When a collision is found, we set the (i,j)th element of the kernel to 1. When there is no collision, we set that element to 0.
We finally compute the intensity value for the vibrator by averaging all the values in the kernel. The result is a number in the range [0, 1] that is used as an intensity multiplier for the corresponding vibrator. The intensity values are updated on every frame and sent to the microcontroller through the application.

System Trial
As a proof of concept, the system was tested by performing a simple task of verifying whether users could distinguish the shape of the objects through touch alone utilizing the described system.
Once the users were fitted with the vibrotactile glove, the system was initialized and the users assumed the Kinect calibration pose for the system to initialize the tracking ( Figure  6). Next, they were shown the virtual environment thro LCD screen. The virtual environment, as described earlier, consisted of the outlines of four basic shapes: a circle, a cross, a square and a triangle, as well as the virtual representation of the ). Once they had obtained a sense of the scale and proportion of the virtual objects compared to their own hand (as represented by the hand icon), the display was turned away from them.
One of the four virtual objects was then placed in front of the user users could then start exploring the space in front of them, in order to try to figure out the shape which they were presented with. The task would finish once the users had guessed the correct shape.
The positional data of the hand were recorded through the application in an external file and the resulting graphs were plotted using Matlab. In order to portray the evolution of the the color of the line changes from black to red, where black indicates the onset of the trial and moves on to red at the final stages of the exploration and the finalization of the trial.
The results obtained from 10 trials between two users (5 trials per user), are shown in Figure 7

III. DISCUSSION
In the following subsection we discuss the system trials we carried out to show how the system operates and to illustrate some of its possibilities. The next subsection deals with the limitations of the current system and possible improvements. The third and fourth sections talk about future work regarding the system and potential applications for the near future.

System Trials
From observing the graphs provided in Figure 7, we can see a strong correlation between the shape of the virtual objects and some of the trajectories (trials 1, 3, 4,5,8,9,10) followed by the Furthermore, we can see that as time progresses, i.e. as the line becomes redder, the movements of the users become more refined and closer to the actual shape of the virtual object (for example in trial No10, it can be seen that during the final stages of the task, just before figuring out the shape, the user once having explored the entirety of the shape, spends more time in exploring finer details such as the upper left corner of the cross shaped object).

Limitations and Improvements
Throughout the testing of our system we observed a number of limitations regarding different aspects of the system and ways of overcoming them or improving them in future versions.
For the haptic glove, the limitations involved the low haptic resolution due to the number of actuators, the size of the actuators and the diffusion and propagation of the stimulus across the glove fabric. To overcome these problems, a future version of the haptic glove could include a larger number of smaller actuators (e.g. 8mm in diameter), for example in an arrangement of 3 per finger and 9 for the palm totaling 24 vibrators. This would increase considerably the haptic resolution and ability to sense finer detail than the current configuration. Also, a version of the glove where the vibrators are placed inside the glove so as to be in direct contact with the propagation.
Regarding the usage of Kinect as a tracking system, although very easy to setup and implement, does not provide very accurate tracking for this application, resulting in some jittering of the hand position, which in turn also may provide erroneous haptic feedback and confuse the user as for the shape of the virtual object. This could be significantly improved with the usage of inertial or marker-based optical tracking systems which would enhance tracking stability.

Future Work
We plan to evolve our current system by improving its functionality and enhancing its capabilities. Here, we present our ideas on these improvements.
At present only positions are being read from the Kinect. In the future we will include also rotations so that users can explore virtual objects not only by moving but also by changing the orientation of their hand. We believe that this will result in a better and more immersive exploration of the VE.
We will also improve our work by using a tracking system more stable than the Kinect to get accurate translations and rotations. That would be the case when using mature technologies like inertial or marker-based optical tracking systems.
Having established a sufficiently stable tracking solution as well as tracking of the position and rotation of the hand, we will include depth sensing as well to explore three-dimensional virtual objects.
In the future we will also conduct a user study with subjects, to find an optimal configuration for our system variables. In particular, we will study how the parameters of virtual hand size, number of vibrators, size of vibrators and kernel size would affect the exploration of the environment.
Finally, we plan to introduce finger tracking and couple it explore a VE from an avatar-embodied first-person perspective through a head mounted display.

Potential Applications
A recurrent problem in VR applications is the lack of physicality when exploring the VE. In many immersive VR applications it is necessary for users to feel presence [12]. When users try to interact with a virtual object by touching it and the environment fails in providing the corresponding haptic response, there is a break in presence [12] and the illusion of being in the VE is diminished. We believe our system could be PHOTO PLACE used to add physicality to a VE. For example, our application could easily be adapted to provide users with two haptic gloves and head tracking. Using both arms could potentially enhance the exploration of the environment by providing users with additional sensitive clues not only coming from the vibrotactile devices themselves but also from proprioceptive information from the position of both arms.
In the context of VR interaction systems, our work could also be used to interact with objects in an immersive virtual environment. The collision system could be used to allow users reach.
In the long term, we think that such a system could be potentially used to help people navigate around physical obstacles in the real world when the sense of vision is not sufficient. Depth cameras like the Kinect could input distance estimation to objects while the vibrotactile system would output stimuli we could use as a guide. It could be used as a sensory substitution from one modality into the haptic one.

IV. CONCLUSION
We have developed a low-cost system for interaction with a VE that uses a prototype vibrotactile glove in conjunction with a Kinect camera to perform the tracking and allows the user to explore virtual objects in a VE. As a test of the system, we devised a task where the user had to understand the shape of the object through touch alone and the results were very well correlated with the expected outcome. The trajectory of the was placed in front of them.
From these results we also identified several limitations to our system as well as solutions to overcome them in future versions.
Overall this system provides a very good starting point to adding an element of physicality within a VE and creating a more immersive experience for the user.