“Real Baby-Real Family”- Multi-Sensory Feedback Tangible Baby VR

“Real Baby Real Family” is a Virtual Reality baby nursery simulator that aims at conveying the parental and family love to the audiences. It has since been exhibited at Laval Virtual 2017, Anime Expo 2017, and SIGGRAPH 2017. This article will attempt to discuss the feedback and reviews gathered during the international exhibitions and shed light on the future development of this project. Index Terms Virtual Reality; Face Generation; Nursing Simulator Environment; Haptic Feedback; Baby.


Introduction
Babies are the fruits of parental love. Such "conjugal love" and "love in family ties" can be used as a synonym for a child, or a baby. This project, named "Real Baby -Real Family"(MOCHIZUKI and NISHIKIZAWA ), is a virtual reality baby nursery simulator that allows friends, homosexual couples, and heterosexual couples to experience the joy of raising a child. It is our hope to invoke deep discussions about family ties and love with this VR project. In this paper, the team reports the motivation and concept of this project in its initial phases during the 24th International Collegiate Virtual Reality Contest (IVRC 2016) and the different iterations during the many international exhibitions such as Laval Virtual 2017, Anime Expo 2017, and SIGGRAPH 2017. Reactions and feedback from the attendees as well as future possibilities of this baby nursery simulator are discussed in detail.

Related Works
Below are some of the previous researches that also uses baby as their subject matter. Neuro-Baby (TOSA 1995 YOTARO (ONO et al. 2009). Amongst these researches, YOTARO can be considered as a work that manifests a realistic baby. YOTARO is a baby robot that has a runny nose and many different emotions. However, while YOTARO can be considered as a precedent which represents a realistic VR baby robot, it's face is that of an illustration and not a real human being, making it look as if the baby is not related to the players and is instead a fictional character. "Real Baby -E-mail: titi@tutu.com Real Family" generates a baby from facial images of the heterosexual or homosexual couples playing it thus those experiencing this project get a baby looking just like the themselves. According to the "Facial aging simulation based on facial domain knowledge"(ISONO, HASHIMOTO, and HORI 2004) of Isono et al., by combining the averaged face and an actual childhood photo, it is possible to achieve the removal of wrinkles and age spots while keeping the characteristics of the player. After going through further treatments to smoothing the skin area, it is possible to generate age reduction images.
Design "Real Baby -Real Family" is characterized by a virtual family enabling the players to communicate with a baby through haptic, visual, and audio feedback while wearing a headmounted display (HMD). According to our hypothesis, creating a believable baby looking like our players will make people more emotionally invested with the experience. Thus we created a baby generated from players' photographs. The most important part of our project; however, is that we constructed a physical baby that can interact with the audience. Projects such as YOTARO features a lot of interactivity but none allows the player to interact with the baby through haptic means. This means one cannot hug the baby which is a fundamental interaction in raising a baby. On the other hand, projects such as Laerdal's high performance infant medical training simulator "SimBaby TM " and "SimNewB R "(Corporation ) focuses on treatment and is too expensive for the general public. The biggest difference between baby simulator in the medical and VR world is those developed for medical purpose aims at patient treatment while VR baby simulator aims at creating a believable experience that touches the hearts of the players. "Real Baby -Real Family" is unique for allowing players to (1) hug and interact with the baby physically, (2) creating a baby looking just like the players, (3) contains audio components.

Holdable baby device
In order to hug a baby doll without experiencing discomfort while wearing the HMD, there is a need for high precision position tracking of the baby doll. However, this is hindered by image recognition ability of a one eyed camera therefore slowing down the position tracking speed. Our proposed method uses multipoint image sensor implemented in the controller of HTC Vive to enable high precision position tracking. This allows players to hug the baby without getting visually disoriented ( fig.1).

Visual Face Generator
Below are three characteristics of our baby face generator: (1) Generating a face fitting the players' skin color, (2) Creating the baby's face from multiple photographs, (3) Reversing the age of the players to fit that of the baby.
Average Baby Face Generation It determines the skin color of the baby by averaging the skin tone of the two photographs. Our system also morphs the colors and shapes of the faces using OpenCV. Lastly it collects 16 baby face images using Japanese Google Image Search.
Get Face Landmark Index In order to morph the many photographs, baby images, and corresponding points in the feature points, it is necessary to obtain the index. In the proposed method, we use the Dlib of open source library, with results that have been learned by the data set iBUG-300-W, to get the points of each part of the face. As a result of fig.2, the eyes, noses, mouths, eyebrows, the feature points, and the 68 points with index composed of contour, are realized by automation morphing. Here it can be seen that morphing are concentrated in the mouth and even parts of the face. Inverse Age Progression Inverse age progression is the process of generating a final image from the averaged age of the facial images at Get Face Landmark Index. This is all done without collapsing baby face images obtained in Average Baby Face Generation. The implementation uses a Dlib and OpenCV. It obtains feature points obtained by Dlib, performs image generation by passing the coordinate data of the feature points OpenCV side. Dlib automatically sorts the feature points obtained and treated them as a landmark. Its index is unchanged in all of the facial images. By setting the contribution ratio with respect to the index number, it is possible to process each selective parts. The color components are produced by Average Baby Face Generation, the present process is a concept that only features the specified parts inherited. If carried out well as deformation of 3D modeling, contour around the chin and other aspects should also be considered. The generated facial image is used by Live2D.

Display of Haptics Synchronized with Voice
When holding the baby doll, the doll is strengthening the interaction by vibro of Vibro transducer Vp2 (Vp210) which was mounted on the baby doll. This vibro is controlled by voice. Only by passing the band-pass filter using a Fourier transform effective frequency band (5Hz 200Hz) and extracted as the vibration data onto an output audio can it perform real-time tactile presentation through HMD attendant controller (Vive Controller). However, since it is the only vibrator that is built in the standard Vive controller, we felt its electric current output which vibrates the baby doll was insufficient after many exhibitions. To fix this problem we mounted the Vp210 onto Vive controller to improve the oscillating capability.

Content of Experience
This work allows for single player mode and two player collaboration. By making the photo of the players look younger and then combine the photographs together, the baby's face was born. Through taking care of the generated baby, players can experience the importance of parental love and family ties with this VR project ( fig.3). During the virtual world, one of the the Vive controllers take on the shape of a milk bottle for the baby. Sensory clues such as the increase in milk consumption by the baby and a sense of touch have all been implemented. After the experience, a copy of the mother's pocket book contains the image of the baby's face was issued to the player as a souvenir. Fig.4 is a block diagram of the system for generating a baby's facial textures in real time. The essence of this work is to generate a texture of baby face that changes according to the players' input image. Because of this, there is a major difference between our achieved research result and the below commonly used animation methods: Using the frame by frame animation technique that animates objects using subtle differences between each image and changing facial Figure 3: Picture of experience expression by changing the location of vertices of 3D models. Therefore, We implemented real-time facial expression changes by the use of RenderTexture which is one of Live2D and Unity features. In preparing for vertex deformation data for use in Unity In Live2D, it was prepared as follows: Material texture to be used as the face of the template players are using (right eye, left eye, mouth, and foundation). Read a file that Live2DManager has generated by the inverse age progression first on the Unity side, to draw using a method called the Graphics class DrawMeshNow. The texture was then rendered using a camera component drawing that have been Live2DModelUnity. Finally, the rendered texture was used to set the material of the face of the baby 3D model to the texture. The different states of the baby model: cry, drink milk, and sleep are monitored by BabyEmotion to manipulate the Animator of animation state. Furthermore, in the Animator, the animation state was operated by parameters of Live2D, then the facial expression was changed. Live2DModelUnity's drawing, the rendering of RenderTexture, a Unity Reference, can change the texture in real time because it is performed every frame. It is generating the visual texture of baby face image in real time by the above operation.

Interaction using image, voice and haptic
Fig5 is a diagram that presents baby voices and vibro tactile associated with it. BabyEmotion reads wave files such as a baby's voice, which was prepared in advance from the audio folder, including: moody, laugh, cry, start drinking, stop drinking, waiting, and sleeping. This BabyEmotion sends the audio files corresponding to the parameter information to AudioSource where the audio file is outputted to a headphone via the HMD as the audio information is presented as haptic. Sampling period in this case is the same as drawing updated frequency (or 100 FPS). Vibration data is what tells TriggerHapticPulse how much should the Vive-Controller vibrate. TriggerHapticPulse is a function within SteamVR Controller which in term is a function within SteamVR Plugin. This makes the vibrator controllable by Vive controller via the speech waveform. In parallel with the process of the spectrum of the voice data are WaveGetVibration and SteamVR Plugin located inside Unity. In order to vibrate the controller mounted onto baby, WaveGetVibration uses SteamVR Controller. After that, it gives the spectrum extracted from the above-mentioned audio data to the argument durationMicroSec of TriggerHapticPulse which is a method of SteamVR Controller. The above processing is performed at the same frequency as the drawing update. In addition, while performing a process of the spectra of audio data, the process of giving the spectrum to the controller was at the same sampling period. Accordingly, the vibration waveform is to be reproduced from speech waveform and Vive controller to be reproduced from the headphones, approximately becomes the same waveform. On the other hand, the parameter information of BabyEmotion (are drinking milk, milk strikes in addition to mouth) is the event the baby is utteied in response to.

Results Obtained from Exhibitions
Feedbacks for Baby Face Generation Although this system is capable of creating a baby's facial image that is similar to the players', in the questionnaire, which recorded the responses of 12 people comprised of both game developers and students, with "similar" on one end, "not similare" on the other end, and were asked to evaluate in four stages with the maximum score being 4 points symbolizing most similar. The average score was 2.27 points which can still be improved. Furthermore, it is necessary to generate a baby's facial image of different races especially for international exhibitions in the future.

Haptic Feedback
Compared to the standard installed vibrator Vive controller, although the Vive controller fitted with a Vp210 gives out a stronger vibration, if players shift their positions or change their postures while holding the baby, our system will have a hard time adjusting to the change. For future challenges we plan on enhancing the baby mockup.

Possibility of Nursing Simulator using VR
In recent years, the nursing educations in Japan, in order to foster nurses with practical skills that can support a diverse of clinical practices, are looking into simulation educational systems such as the objective structured clinical examination (OSCE). The practical simulator nursing education using virtual patients has attracted much attention. Our Virtual Reality work, "Real Baby -Real Family", as hinted by its name, was implemented using a realistic baby and is accessible to the general public where all can experience the love of a new founded family. This type of VR project remains unrealized until now where we constructed a virtual baby that inherits facial features of the players, have a tactile sensation, and allows new and inexperienced parents to quickly become accustomed with raising a child. For future challenges, we hope to simulate other essential actions relating to raising a child such as breast feeding as well as giving the virtual baby a heartbeat and body temperature.

Application to Entertainment System
When exhibited at IVRC 2016, this VR system has an option that allows players to create a two-dimensional image of the baby from actual photographs of the players or substitution photos. This Virtual Reality system gives the illusion that The believable 3D baby model inside the virtual world and the 2D baby photo created from the players' image further enhances the experience. In the future we would like to tailor this entertainment system to the subculture market that makes two-dimensional images and babies.

International Exhibitions
Since its inception in IVRC 2016, "Real Baby -Real Family" has participated in three international exhibitions thus far, and these being Laval Virtual 2017, Anime Expo 2017, and SIGGRAPH 2017. During the exhibitions, we have obtained numerous feedback and reviews from users of different nationalities and origins. This section will explain the exhibition findings.

Laval Virtual 2017
Laval Virtual 2017 in France is the first opportunity to exhibit "Real Baby -Real Family" on the international stage. From March 22nd to March 26th 2017, the Baby team exhibited at Laval Virtual and the project was played a total of 292 times. A baby racial selection system was implemented for this exhibition featuring three types of skin tones for the virtual reality baby face: Caucasian, Asian, and African. The rate of which each of the skin tone is chosen is roughly 30 percent for each. Attendees' impressions and feedback are generally positive, with many people praising the experience for its concept, realism, and potential in the future. It would also appear that the French audience understands the underlying goal and message of the project clearly, which is to tackle the issue of declining birthrate and rapidly aging society in Japan.

Anime Expo 2017
Anime Expo 2017 which goes from July 1st to July 4th is the second international event "Real Baby -Real Family" participated this year. The exposition was held at the Los Angeles Convention Center and the Baby project was played for a total of 349 times. The attendees are mostly cosplayers or anime enthusiasts with some professionals working in the anime and manga industry. This exhibiting opportunity was given by Shueisha, a world famous manga publisher in Japan, in order to promote Japanese manga culture and virtual reality projects. Exhibiting together with "Real Baby -Real Family" are other virtual reality projects from the Tokyo VR Startup. "Real Baby -Real Family" receives mixed reviews at Anime Expo 2017, with the majority of the players think of the baby as either creepy or interesting. It is worth noting however, that the young female attendees have taken a liking at the project. Also worth noting that this is the first time "Real Baby -Real Family" has employed the MasQueRade, a QR code based online evaluation system developed by Shirai-lab of Kanagawa Institute of Technology to conduct surveys on the attendees. The figure below is a picture of MasQueRade attached to the VR hygiene masks at Anime Expo 2017.

SIGGRAPH 2017
SIGGRAPH 2017 is the last and biggest conference "Real baby -Real Family" has attended thus far. From July 30th to August 3rd, the baby project was exhibited at the Emerg- ing Technology section of SIGGRAPH along with 28 other projects. The majority of attendees differs from that of the previous exhibitions in that they are professionals and researchers in the computer graphics world. In addition, most of the attendees are single and do not possess much experiences handling a baby. The feedback and results "Real Baby -Real Family" received during SIGGRAPH 2017 is much better than that of the Anime Expo 2017 with many attendees recognizing the potentials and future applications of this project. While some attendees have remarked that the baby faces of the same race all look the same and that having a HTC VIVE controller sticking out of the baby's head makes it look kind of creepy, others have praised the project for its realistic baby weight and believable interactions. Other comments given to us by our attendees include the desire to use the baby face generation software in sperm bank in order to predict the future facial features of the infants from different sperm donors.
Aside from attendees' comments the Baby team has also conducted surveys regarding to their opinions on the different kinds of babies they would like to have play with. One of the questions being "Which gender of the baby do you want to see?" and below is the graph that showcases the answers obtained. Figure 7: Answers obtained from players before the experience using the MasQueRade system regarding their opinions on the baby's gender they would like to play with.
Aside from the baby's gender, the baby team also asks the type of personality they would like their baby to play with. The graph below shows data gathered regarding the players' choice on the above question.
Based on the data above the team can determine that the players really appreciate more varieties and interactions for the babies they get to interact with. In the future the "Real Baby -Real Family" team would like to use the information obtained to further improve the experience for the players and offer more variety in the experience. All attendees were asked "Do you want to have a baby?" before and after the experience during the Laval Virtual, Anime Expo, and SIGGRAPH exhibition. Figure 9 showcases Laval Virtual 2017 attendees' opinions regarding whether they want to have a baby or not before and after the experience. As we can see, 19% of total players changed Figure 9: Players' response to whether they would like to have a baby before and after the experience. A refers to answering "Yes" before and after the experience. B refers to answering "No" and then "Yes" before and after the experience. C refers to answering "Yes" and then "No" before and after, while D stands for "No" and "No" their answers from not wanting to have a baby to wanting a baby after the experience while 15% of total players changed their answers from wanting a baby to not wanting a baby after the experience. The rest 66% of the players' opinions remained unchanged before and after the experience. Based on this data collected it is safe to say that "Real Baby -Real Family" is not only well received by the French and European audience, but also has a positive impact when it comes to convincing them to want to have children. Also note that the majority of people (75%) who wants to have kids stayed wanting to have kids while roughly half of the attendees (50%) who answered they do not want to have their baby changed their answers to wanting a baby after playing the experience. Figure 10 represents the players' answers to the questions for Anime Expo 2017. Unlike Laval Virtual, this time the data was obtained using a QR Code Questionarie system named MasQueRade developed by Shirai-lab. In compari- Figure 10: A refers to answering "Yes" before and after the experience. B refers to answering "No" and then "Yes" before and after the experience. C refers to answering "Yes" and then "No" before and after, while D stands for "No" and "No" son to Laval Virtual, there is a much greater number of people who changes from wanting to have baby to not wanting to have baby after the experience (108 attendees or roughly 31% of total players) compare to the attendees who changed to not wanting to have a baby to wanting to have a baby (30 attendees, or roughly 9% of total players). Also note that the 108 out of 275, or 39% of players who answered they wanted to have baby before the experience changed their answer to no afterwards whereas only 30 out of 74 players, or 41% of players changed their answer from "No" to "Yes" after the experience. While there is a much larger pool of attendees who answered yes they wanted to have a baby in the first place compare to those who answered no they do not want a baby before the experience and that the percentage of people who changes their opinions after the experience is roughly the same, there is still more than 3 times of people who changed from "Yes" to "No" compare to "No" to "Yes". Figure 11 showcases the attendees' answers to the questions at SIGGRAPH 2017. The data collected from SIG- Figure 11: A refers to answering "Yes" before and after the experience. B refers to answering "No" and then "Yes" before and after the experience. C refers to answering "Yes" and then "No" before and after the experience, while D stands for "No" and "No" GRAPH shows an almost equal number of people changing their opinion from not wanting a baby to wanting a baby at 15% of total players and wanting a baby to not wanting a baby at 16% of total players. Number and percentage of players who changed from "Yes" to "No" after the experience is 42 out of 180 players or 23% while players who changed from "No" to "Yes" stands at 42 out of 92 or 46%.
Despite both SIGGRAPH and Anime Expo both takes place in Los Angeles Convention Center, "Real Baby -Real Family" is more effective at convincing SIGGRAPH attendees to want to have baby than Anime Expo attendees despite of the version shown in both conferences being exactly the same. This can be contributed to a variety of reasons ranging from SIGGRAPH attendees coming from the scientific community thus are more acceptable to new systems whereas attendees of Anime Expo are expecting entertainment when entering the conference therefore are not prepared for the technology "Real Baby -Real Family" has to offer thus reacts more negatively towards it. It would also appear that French attendees are more likely to get persuaded by "Real Baby -Real Family" to have baby compared to their American counterparts.

Conclusion
Throughout the IVRC 2016 Tokyo qualifying round and later the final competition, the baby team continued to refine the system until it delivers a satisfying baby themed experience. Using the knowledge gained from exhibitions, the weight and intensity of vibration of the baby doll were adjusted to deliver a believable sense of touch when interacting with the baby. Furthermore, we succeeded in creating a baby face that borrows features from the players' photographs and created a virtual baby capable of fostering deeper connections with the players than previous works such as YOTARO. We also experimented with a nursing simulator child care technique in this project and the result showed the potential of a nursing simulator technique for VR system. In the future we are hoping to create more VR entertainment systems featuring a complete VR world and real life physical objects such as 2D picture and baby doll. The system will be updated to develop educational opportunities and it also develop the new media service like current photo media for family relationships.