Digital Analysis and Visualization of Swimming Motion

—Competitive swimming is a demanding sport that requires rigorous training to achieve technical perfection. Research on computer simulations of flow have improved our understanding of how thrust and drag can be optimized for better performance. However, for a swimmer translating this information to technical improvement can be difficult. In this paper, we present an analysis and visualization framework for swimming motion that uses virtual reality to display 3-dimensional models of swimmers. The system allows users to digitize their motions from video sequences, create personalized virtual representations by morphing prototypical polygonal models, visualizing motion characteristics and comparing their motions to other competitors stored in a library. The use of virtual reality alleviates many problems associated by the current video-based visualization methods for analyzing swimming motion. 1


INTRODUCTION
Swimming as a sport requires years of rigorous training to develop physical capacity as well as to achieve technical perfection.Competitive swimming can hinge upon fractions of seconds to decide the winners, which has motivated research efforts to take advantage of computer simulations studying swimming motion to improve the athletes' performance.However, research on swimming motion has usually focused on the analysis of the fluid flow around the swimmer.Significant advances in the computational fluid dynamics (CFD) has enabled simulations to help manufacture efficient swimsuits [1] and better understand the mechanisms of swimming motion [2].As useful and important these research efforts are, they neglect to address some fundamental issues.Even though swimming is a widely popular sport and seeing an accurate computational fluid dynamics simulation of a world-class swimmer could help educate a swimming coach or an up-and-coming young swimmer, these effects would be improved if this process can be individualized.As these simulations and visualizations usually require high-performance supercomputers, trained experts and Manuscript Received on 19 April 2011.Email: kirmizi@gwmail.gwu.eduexpensive software to be correctly run, it is unrealistic to assume that they can be accessible to a wide range of amateur or professional athletes.Another important issue is that these simulation results can be very beneficial to a swimsuit manufacturer or fluid dynamics expert, while a coach would be more interested in seeing if a swimmer's stroke is technically sound.Traditionally, these analyses have been conducted by using videos taken from several principal viewpoints, but problems present in using 2D videos in general (e.g.limited viewpoints, loss of 3D information) and inherent in swimming (e.g.occluders such as bubbles) make this task more difficult.
The visualization approach presented in this paper is motivated by the problems mentioned above.We aim to provide end users an easy to use system where they can use information about individual swimmers to transform prototypical swimmer models and compare their motions to other swimmers stored in a library; be it those of elite competitors or their teammates.Since there are many commercial systems available for flow visualization and analysis, the emphasis in our research is given to acquisition, visualization and comparison of motion rather than flow.The introduced approach allows the acquisition of motion from videos, storage of these motions in a library, morphing prototypical swimmer models for analysis (and possibly for fluid dynamics simulation [3]) and comparison using an interactive 3D visualization approach.The 3D visualization is augmented with tools to show information about speed and acceleration of joints or user selected points on the swimmer using numerical outputs, traces and graphs.
The paper is organized as follows.In Section 2, related work about swimming and sports motion analysis is discussed.Section 3 describes our motion acquisition approach.In Section 4, we introduce the methodology used to morph prototypical swimmer models for analysis and visualization.Section 5 presents the visualization approach used.We conclude with a discussion about these approaches and future work in Section 6.

II. RELATED WORK
In simplest terms, a swimmer needs to maximize thrust and minimize drag to swim faster.Toussaint et al. [4] give an excellent overview of biomechanics involved in swimming.However, thrust and drag can be hard to measure in an aquatic

Digital Analysis and Visualization of Swimming Motion
Can Kirmizibayrak 1 , Jean Honorio 2 , Xiaolong Jiang 1 , Russell Mark 3 and James K. Hahn  [2] for an overview.Some examples of simulations include the dolphin kick [3], using particle hydrodynamics [5], or simulating and visualizing flow around the body surface ([6], [7]) .For a swimmer, these kinds of studies are limited in their flexibility and interactivity since they are computationally expensive and usually require specialized software.
For analyzing sports motion in general, there are mainly two methods.First method is based on video analysis, where improvements can be made by comparing the videos of the trainees with those of elite athletes.Lok and Chan [8] developed a model-based human motion analysis system that can track human movement in monocular image sequence with minimum constraint without markers or sensors attached to the subject.Given a clip of the video, they first manually fit the 3D human model to the subject in the first frame of the video; then do background subtraction, silhouette extraction and discrete Kalman filtering to predict the pose of the subject in each image frame.Saito et al. [9] introduced methods for sports scene analysis and visualization by integrating the tracking data from multiple videos captured with multiple cameras.They also present a method for free-viewpoint visualization of a soccer game as a technique for visualizing the soccer scene by view interpolation between real cameras near the virtual viewpoint at each frame.Additionally, there are commercial systems available for video analysis such as Dartfish [10] and Sports Motion [11].The common drawback of video-based systems is they limit the user to the viewpoint of the camera.This is even more troublesome in swimming, where the video can contain artifacts due to reflection and refraction and contain occluders due to splashes and bubbles.Furthermore, filming underwater is more restrictive and is usually limited to cameras whose locations are mechanically controlled.
The alternative approach is based on virtual reality, where the athlete can learn and improve performance mainly through interaction with the virtual environment.Pingali et al. [12] use motion trajectories and heat maps which are obtained by digitizing tennis ball motion.Bideau et al. [13] use Virtual Reality to study the perception-action loop in athletes: how perception influences choices about which action to perform, and how those choices influence subsequent perception.They design a framework that uses video-game technology, including a sophisticated animation engine and use this framework to conduct two case studies: The first was a perception-only task to evaluate rugby players' ability to detect deceptive movement.
The second concerned a perception-action task to analyze handball goalkeepers' responses (action) when facing different ball trajectories (perception).These case studies demonstrate the advantages of using VR to better understand the perception-action loop and thus to analyze sports performance.Motion capture in general has become very important in this sense to transfer the motions of the athletes to the virtual world, but unfortunately is of limited use in our application domain because of the inherent difficulties caused by the swimming motion being in water.
Our work follows the virtual reality approach and introduces methods to adapt the current approaches to the swimming domain.Our goal is to create an intuitive and easy to use framework where swimmers can individualize the process to visualize their own motions in 3D, and make comparisons to other athletes of varying ability levels.We have chosen to focus on visualization of swimming motion rather than the flow itself since this to our knowledge has not been researched thoroughly before.However, flow visualization can easily be incorporated to our framework since we use standard OpenGL rendering methods.Furthermore, our human body parameterization approach makes this process easier by providing an athlete-specific model based on measurements, which can be used to perform CFD simulations.

III. MOTION CAPTURE USING SYNCHRONIZED VIDEO SEQUENCES
Although different types of motion capture systems exist already, they cannot be used to capture motion under water.Optical systems cannot be used due to the interference (reflection and refraction, water bubbles, etc.) from the water and electromagnetics based systems are not water resistant and may interfere with the swimming motion.Thus, there is a need for a system that can accurately capture motion of swimmers.Through a motion storage module, digitized motion can be stored in online databases and can be used to analyze and visualize swimmers in 3D.
The module provides a semi-automated system for the user to mark the location of the limbs on a number of frames in the video.The video will accompany a still image of the swimmer standing in a T position in order to calculate the lengths of the limbs.A prototypical articulated figure is then overlaid on top of the video frame, allowing the user to see the positions of the limbs.The module outputs 3D motion data that correspond to the video.This data can then be stored using the storage module and be viewed by the visualization and analysis module.Since one of the most common types of video is one taken from the side (e.g.Fig. 1, Image B), we started with the assumption that this is the only footage available and implemented the system accordingly.The addition of the other orthogonal synchronized viewpoints can improve the acquisition of a more robust and accurate motion.In order to determine the relative lengths of the limbs, a still image of the standing swimmer in a T-pose is taken from the front.The user first clicks on the positions of all the joints in the still image.Using these joint positions, the lengths of each bone is calculated.The user then clicks on the joints on each of the video frames.
The lengths of the limbs are used to calculate the z-coordinate of the joint position.Since the joint positions provided by the user are in two dimensions (x and y coordinates), the third dimension (z) needs to be calculated by the system.This is done by calculating the ratio of the limb lengths from the still image and the length visible in the 2D video image.Since we know the projected limb length in the image (p) and the real length measured before (r), the depth (d) can be calculated using simple trigonometry with (1): This has to done in a hierarchical manner where depth of each joint in the skeleton is calculated relative to its parent joint.The only missing piece of information left is the orientation of the limb -whether it is going into the screen or coming out.This can be either provided by the user, or derived from the second synchronized video sequence.Once the joint positions are calculated, they are converted into joint angles for each joint.Vectors defining the orientation of each of the two bones are first calculated using the joint positions.The dot product of these two vectors then gives us the angles between them.A video sequence of a breaststroke was used to test the system.The user then selected a few frames from the video and defined the joint positions in the video.The resulting digitized motion data is shown in Fig. 2.

IV. HUMAN BODY SHAPE PARAMETRIZATION USING STANDARDIZED ANTHROPOMETRIC MEASURES
For effective visualization and computer simulations, we need an accurate representation of the swimmer's body.Range scanning allows extraction of very detailed human body shapes.However, several artifacts are also introduced such as noisy parts and holes.Substantial effort is needed in order to convert these geometries into models suitable for animation, fluid dynamics simulation and motion parameterization.While animation requires information about inner structures (e.g.skeleton); motion parameterization additionally requires the mapping of specific anthropometric measurements (e.g.shoulder-elbow length) into the resulting models.Furthermore, the range scanning process is expensive and is not likely to be available to many athletes.In order to alleviate these problems, we propose to morph a prototypical human body mesh into particular swimmers.
Prior research has focused on face deformation for different goals, such as: automatic face generation from anthropometric measurements [14], facial reconstruction for postmortem identification of humans from their skeletal remains [15], as well as growth and aging simulation [16].Our method parameterizes human body shapes by using standardized anthropometric measurements.Three different control layers provide a method for easily generating several shapes with the input of: (i) weight and stature, (ii) weight and bone lengths (i.e.weight, stature, sitting height, biacromial breadth, bitrochanteric breadth, upper arm length, fore arm length, hand length, thigh length, calf length, foot length), or (iii) all the measurements.For the coarse control layers, the remaining measurements are calculated by linear regression using the 1988 Anthropometric Survey of US Army Personnel.

Our model includes three measurement categories [17]: (i)
Euclidean distances between two landmarks, (ii) axial distances between two landmarks with respect to an axis, and (iii) tangential distances, i.e. the distance between two  Ankle circumference t Foot length a landmarks over the surface of the skin.Table I shows the thirty three anthropometric measurements included in our model as they explain most of the representative changes in mass [17], [18]: The anthropometric measurements are mapped into a set of linear constraints among 114 landmarks, which are well-defined features over the skin, usually with respect to bone structures.For instance, the upper arm length is mapped as an axial distance between the Olecranon and the Acromiale, and the wrist breadth is mapped as a Euclidean distance between the Radial Styloid and Ulnar Styloid.
Fig. 3 shows our three constraint categories.An axial constraint is given as a goal distance r between two unknown landmark coordinates ti and tj with respect to a principal axis, usually Y.This is a linear constraint, since only one coordinate (either X, Y or Z) needs to be modified in order to reach the goal distance.A Euclidean constraint is given as a goal distance r between two unknown landmark coordinates ti and tj.We linearize this constraint by generating one axial constraint for each of the three principal axes.Finally, a tangential constraint is given as a goal distance r between two unknown landmark coordinates ti and tj.We approximate this constraint as a set of Euclidean constraints between the m vertices in the shortest path from ti to tj in the prototypical mesh.Given that the above constraints were previously linearized, we iteratively solve an ordinary least squares problem as in [16] until a small error is reached.Thus, we compute a new set of 114 anthropometric landmark coordinates that best meet the anthropometric measurements of a particular person.
The prototypical mesh is then morphed by using radial basis function (RBF) deformation with a biharmonic spline as in [15,16].The RBF function maps the original 114 landmarks to the ones computed above, and allows deforming every vertex in the prototypical mesh as well as the skeletal structure.Results of our system are shown in Fig. 4.
The resulting mesh has an accurate representation of the skeleton, which is important for the visualization and comparison of different swimmers; and an accurate representation of the body surface, which can be crucial for fluid dynamics simulations.

V. VISUALIZATION AND ANALYSIS OF SWIMMING MOTION
In Sections 3 and 4, we have discussed methods to acquire the motion and to transform polygonal models to match the body types and measurements of the desired swimmer.The last step in our approach is presenting this information to the user.We have implemented an interactive visualization system that offers various visual and numerical analysis tools.
We start with the polygonal mesh that has been deformed to fit the skeletal and pose information acquired during preceding steps.In other words, we use a model where the skeleton 'drives' the mesh.There is a wide body of literature and a number of commercial software solutions for mesh deformation, a subject beyond the scope of this work.We used Autodesk MotionBuilder™ to animate the polygonal model and saved each resulting frame.Intra-frame animations are done by interpolating in-between vertex positions.This provides us with the animation of a swimmer and the corresponding joint angle values for one or more stroke cycles (depending on the number of cycles acquired in the previous steps An interactive 3D visualization approach has several advantages compared to 2D video-based visualization.First, the user has unlimited control over the viewpoint.This gives the ability to rotate around and zoom to the visualization freely and presenting multiple complementary views consequently (Fig. 5).These multiple views are synchronized and help the understanding of complex motions by displaying possibly occluded regions.
Depending on the information required by the user, the animation can be presented in different ways: using a solid rendering with shading, a vertex rendering mode to visualize the skeleton as well as the swimmer, and a silhouette rendering mode to put more emphasis on the change in the joint angles (Fig. 6).Shadows (or more precisely, projections to X, Y and Z planes) can be added to help the user understand the body motion in 3D from a single view (Fig. 6b).
One of the main advantages of using a skeleton driven animation approach is the presence of numerical information about the swimmer's pose in any temporal location.We take advantage of this by using a variety of visualization techniques.
The first is displaying joint angles in their respective positions (Fig. 7a).In addition to this, the user can click: • Any point on the swimmer and display the current position, velocity and acceleration, • Two points to display the angle and distance between them (Fig. 7b), • Three points to display the angle formed by these three points.
The second analysis tool our visualization system provides is displaying traces of joint positions.This way, the users can see how the joint locations change over time and make necessary improvements to their techniques.A swimmer would gain valuable and unique insight by being able to see the amplitude of their leg kick, the vertical displacement of the hips when they lift their head for a breath, or how much they press their chest downward in butterfly -all aspects that are known to affect performance but are difficult to quantify or visualize.Orthogonal projections of these traces are also provided to improve the understanding of their spatial relationships (Fig. 9).Even though our system provides valuable information about the motion of individual swimmers, the real benefits of these methods are realized by using the presented information for comparison.By using a library of stored motions, we can display multiple swimmers simultaneously (Fig. 8a).Swimming is a very technical sport that requires cyclical and precise motions athletes perfect after intensive training.Since the stored motions in the library are normalized in our acquisition step, swimmers of different capabilities can be displayed together.This can be a powerful comparison tool, which would enable an amateur swimmer to compare his swimming style to an Olympic champion.Even Olympic-level swimmers study video to compare and learn from other Olympians, so the flexibility in comparing techniques and ease of seeing the sometimes subtle differences is critical for an analysis tool to be practical and widely used.Without normalization, this comparison would be very difficult because the speed of an elite swimmer would no doubt be drastically different from an amateur, making a side-by-side comparison less effective.In the comparison mode, two sets of motion data can be loaded from the motion library.The users can take advantage of any combination of visualization tools described above: for instance, one swimmer can be displayed in shaded rendering mode while the other is shown as a silhouette, with traces added for selected joints to see how the motions differ between them (Fig. 8b).
In summary, our visualization system provides users with various tools to display, analyze and compare the digitized motions of one or more swimmers.The tools presented here were chosen to address the problems in analyzing swimming motion.To our knowledge, this is the first visualization and analysis system designed specifically for swimming.

VI. DISCUSSION AND FUTURE WORK
In this paper, we presented a framework to individualize the analysis of swimming motion.By transferring the captured motion from 2D video to 3D rendering, a more flexible and effective visualization can be achieved.The process enables the athletes of various ability levels to digitize their captured motions, and morph prototypical polygonal models to end up with a visualization tailored to that specific swimmer.These models can then be used to visualize and compare with other motions stored in a library, enabling swimmers to compare their techniques to world-class athletes and finding areas to improve.The system was designed in close collaboration with USA Swimming, the governing body for the US Olympic swimming team, and received very positive responses from swimming coaches.Our aim is that the system should be used by athletes and coaches of various backgrounds.Therefore, special care was taken to ensure the compatibility to a variety of software solutions.The pose information, which drives the polygonal model is stored with Acclaim (amc/asf) pose file standard.This means the users could use standard motion editing software or import motion capture data to our system.We aim to make the visualization software available to the public in the near future.Even though designed for swimming, our visualization system is general enough to be applied to a variety of sports applications where the analysis of subtle changes in a cyclical or repeated motion is important (e.g.cycling, baseball pitches, football throwing motion).The focus of the work presented here was analysis and visualization of motion, as we believed this was an important aspect that has usually been overlooked in swimming research where flow visualization received more attention.However, flow information can be easily incorporated into our visualization framework.Moreover, the morphing methodology introduced here can be used to perform athlete-specific fluid dynamics simulations.
One shortcoming of our system is the somewhat labor intensive process of motion acquisition.However, the specific nature of swimming eliminates the possibility of motion capture.Computer vision based techniques are also challenging because of the artifacts such as water splashing or bubbles that occlude the swimmer in video.Furthermore, since the swimmer's body is partly above and partly below the water surface, a computer vision based system would need multiple cameras that need to be synchronized.Even though such complicated setups are not widely available, we believe fully automating the motion acquisition process would greatly improve the usability of our visualization approach by increasing the number of stored motions in the motion library.

1 1
Department of Computer Science, The George Washington University, USA 2 Department of Computer Science, Stony Brook University, USA3USA Swimming environment and the measurement process can alter the very values we are trying to observe.These difficulties made Computational Fluid Dynamics simulations of swimming motion a very attractive area of research.Recent efforts to study swimming motion has focused on these simulations, interested readers can refer to Marinho et al.