An immersive paradigm to study emotional perception in co-presence through avatars

How to design new forms of on line behavioural and emotional interactions in digital spaces, shared virtual environments? What are the perceptual, behavioural and cognitive processes involved when actions, emotions or information are shared between several co-present users? What are the aesthetic and sociological implications of these new forms of spatial mediations? What is the influence of spatial location and emotional valence of pictures on the capacity to memorize the pictures? For this study of perceptual and cognitive processes involved when sharing and memorizing emotions online in digital spaces" we have brought together a multidisciplinary team of researchers in Art and New Media, Cognitive Sciences and Psychiatry. Our goal is to assess the relevance and limitations of new forms of mediation in onlineshared spaces (also commonly referred to as "virtual worlds"). This paper presents a preliminary account of the paradigm. Here we focus on the influence of copresence on the emotional perception of fine arts in a digital environment. This experience was developed by the "Spatial Media" group of EnsadLab in partnership with the Collège de France, supported and funded by PSL* Research


INTRODUCTION
Through the experiment « Art Distance Sharing », our research question is to evaluate the relevance and the limits of new forms of inter-mediation related to the use of shared virtual environments online (also commonly referred to as "virtual worlds"). We are aiming to understand the cognitive and perceptive process involved when a number of users, who are physically distant but present together in digital spaces, through their avatar, are sharing actions and/or emotions. This paper describes the first experiment conducted as part of our research program: « Perception of shared spaces: Study of perceptive and cognitive processes when sharing emotional experiments in virtual environments online. ». We have previously observed that many gaming and artists communities divert original screenplays of video games and use the gaming platforms as « chat rooms » and communication areas, with which and where they share contents, videos, sounds etc. Our research focuses on how being co-present in a virtual world can affect our emotional perception of works of art. We are aiming to understand how the new forms of intermediations shared in virtual worlds can influence the perception and memorization of the content of these worlds.

Context in digital arts and media
For some years now, new forms of Internet inter-mediations, based on the presence of users in shared digital spaces online (MMORPG or virtual worlds) are emerging. During the last 15 years, the computer games technologies have progressively migrated to the web, giving birth to share digital spaces. In these shared digital spaces, an avatar for each partner is used both as an interface tool and a representation tool. It acts for and « is » the person it represents. It locates the user relative to other users and enables the user to interact and communicate (via text and/or voice). New communities of users have invested these virtual worlds stripped from the gaming screenplay and design. Researchers, artists and trainers now exploit the social potential of digital spaces for sharing information, also using them as creation spaces and teaching spaces. 1 During the past years, under Spatial Media research group at EnsadLab, we have conducted observational studies of artistic experiments in shared digital spaces online. 2 We then could identify specific practices and uses dedicated to these spaces, different from other uses of digitals technologies of communication, namely: social networks (based on creation and implementation of pages and profiles), mass We have identified three characteristics of those media: -Media spaces are perceived by means of action. The perception process implies a bodily dimension of the user. The projection of the user's body in the paradigm -through its avatar-, introduces an engaged dimension in the way the user will research information. As the process locates the exploration in a context (spatial memorization, Ars memoriae), it helps the user with the memorizing of information.

Context in neuroscience
These observations also have stimulated the research of the brain mechanisms, which underlie the astonishing capacity to be "present" but also, embodied in the virtual avatar. Mel Slater and other groups brought evidence that the user is "embodied" with his virtual representation, and projects himself cognitively into this new spatial repository. In addition our paradigm allow a study of social interactions between avatars and their consequences upon the operators themselves both in term of emotion and memory.
Recent studies using virtual avatars in autoscopic or heautoscopic situations have improved our knowledge of the neural basis of "out of body" experiences which were known in Neurology. It is not the aim of this paper to review the extensive literature on these questions but we shall mention a few studies, mainly of the group of Alain Berthoz, of the brain areas involved in the ability to manipulate points of view and the brain mechanisms involved in the use avatars in virtual environments ( See list of references).
Faced with another we have the ability to project ourselves into the other's point of view. Either to try to predict their actions or to share their emotions. The different brain networks involved in empathy have been found and shown to be different from those The capacity to know where another individual is directing attention in space and what he or she is seeing on the current visual scene, i.e. 'visual perspective taking', provides critical information for monitoring social interactions. It is likely a prerequisite to understand another's intentions, actions and emotional reactions, as well as to adapt one's own behaviour to the current situation. Many studies showed that one subject can project himself in a virtual avatar to perform spatial tasks, such as imagining a scene from different points of view (ref 10) and that such a spatial computation involve specific brain areas . (ref 9) Another important spatial feature of social interactions is the distance that people maintain between each other when interacting, which is referred to as "proxemics" as defined by Hall in his book "The Hidden dimension (1966). This distance is permanently regulated according to the level of intimacy, the respective directions of gaze, the social and spatial contexts, etc.
Recent studies suggested that the laws of proxemics also apply to virtual environments and avatars7. Moreover, other studies show that interacting with virtual avatars could entail affective reactions in subjects.

The present project
In the present project we aimed at evaluating the influence of emotional content of pictures on our ability to inhabit a digital environment, to experience and to memorize information. Our questions in this experiment were: Does this faculty of empathy apply when two subjects share the same space at a distance, through avatars? Does this co-presenceoutside of our physical bodies -, have an impact on the quality of perception of the emotional content of picture in the digital space? What kind of information should we most likely memorize?
To attempt to answer these questions, we have built-up a multidisciplinary team associating, Francois Garnier, Film director and teacher researcher in Art and communication, head of the research group EnsadLab / Spatial Media of l'École Nationale Supérieure des Arts Décoratifs -Paris; Alain Berthoz: Emeritus professor. Collège de France. Member of the French Academy of Sciences and Academy of Technologies; and Simon Lambrey: Psychiatrist -Doctor in Cognitive Sciences -Former Clinician in "Hôpitaux de Paris". This.is work is supported and funded by Paris Sciences et Lettre Research University, Program "Investissements d'avenir» (ANR-10-IDEX-0001-02 PSL") This research is part of an "Art and Science" approach. It is based on the observation of the practices of communities of players, users or artists. In response to questionings regarding specific practices, we develop screenplays in digital paradigms and study the behaviors and emotional responses of the human subjects in immersion. This approach requires that we develop technologies, define experimental protocols. We need to set up and validate data capture and analysis devices (proxemics, biometric, behavioral and psychological analysis). In this first experience, our hypothesis is copresence through avatars, of human subjects physically distant, in a tridimensional digital space online, influences either positively or negatively the perception and memorization of information. We asked 51 female subjects to visit a virtual art exhibition as if they were members of a jury critic. Three situations were distinguished according to the presence or the absence of a "second avatar" in the place of exposure: a) absent avatar, b) avatar present avoiding contact with the subject's avatar c) second avatar interacting with the subject, for instance coming to look at the same picture. Our objective was to understand the effect of the co-presence of a second avatar a) on the behavior of the subject in the virtual environment and the emotional experience of its content and b) on the memorization of the content of the environment. Various information were recorded during the whole time the subject visits the environment, such as the location of the subject, in the paradigm as well as relative to the works of art; the interpersonal distances between the subject's avatar and the co-present avatar. Physiological data inform us about the behavior and the emotive state of the subject. The subjects also answered a questionnaire at the end of the visit concerning the pictures that they had seen to test their memory.
The study of these data should enable us to measure if the presence of the other avatar had an impact on the subject. The results will allow us to contemplate new experiments to refine our understanding of the cognitive and psychological modalities implemented in these situations of social interactions in virtual worlds and to consider their applications.

Experimental setup
The subject was physically isolated in a booth, standing in front of a screen exhibiting the paradigm of the exhibition 9. Her avatar appeared to her in the foreground, back on the subject (Case 3rd Person).4 The subject used a joystick to control the movements and the orientation of his avatar. (See Figure 1). From a separated cabin, a manipulator -who had no contact with the subject -, manipulated both avatars: active and passive. The presence of two persons (we shall call them "manipulators") was required: 4 We had planned to use one Oculus device: the subject wears an Oculus headset with stereoscopic subjective view (1st person). He stands before a pupil. He uses one of his hands to control his avatar's movements with a joystick. The opposite hand is equipped with biometric sensors and rests on the pupil. This option had the benefit of immersing the subject deeply. The control of the gaze direction was more flexible, more intuitive and its analysis sharper. However preliminary tests carried out quickly proved that the Oculus device option not functioning. None of the five tested subjects could successfully complete the visit of the twelve rooms of the paradigm. All of them complained about vertigo or discomfort. We believe the origin of these vertigo and discomfort hinges on two combined reasons: a) The first version of the Oculus headset offered a poor quality of image and response time. b) The task assigned to the subject: " watch the works of art " required a stable, sharp and repeated gaze over a long period of time.
Manipulator 1 was in contact with the subject. He welcomed her, have her installed, briefed and assisted her during the training phase. He found her at the end of the visit and had her reply to the questionnaire. Manipulator 2, was not in contact with the subject either not visible to her. He launched the application, manipulated the avatars (passive and active) and saved the data. The manipulators team comprised three student-researchers of Spatial Media Lab: Aurélie Herbert (manipulation manager), Dionysis Zamplaras, and Christine Webster.
The subject faced a screen about 2.50 meters wide, distant of 1.50 meter from him. The subject saw his avatar back on him (3rd person). The subject stood in front of a stand on which a joystick was placed. He uses one of his hands to control his avatar's movements as well as to choose the orientation of his point of view, with a joystick. The opposite hand is equipped with a biometric It rested quietly on the stand so as not to perturb the physiological measures.

Virtual exhibition.
The virtual exhibition was composed of a training zone, and the exhibition area itself including twelve successive virtual and similar rooms. In each of them, four works of art (paintings or photographs) were displayed, each on one of the four walls. The works of art were randomly hung on the virtual walls. A door allowed to go from each room to the next. (See Figure 2 and 3) The video games technology was chosen to implement this exhibition. This approach had numerous advantages: -a reduced cost of implementing. -a familiar production environment, mastered by our student-researchers at Spatial Media EnsadLab. -a great flexibility in use and an open programming environment. This allowed us to control the interface/interaction, the interface/visualizations, as well as the biometric sensors, without major development effort. The management of the databases recording was easy.
The paradigm has been modeled in Blender by Donatien Aubert. The entire management of interactions, interfaces and databases was carried out in Unity by Dimitrios Batras, Dionysis Zamplaras. The platform implemented during the project has proven to be flexible to adapt during the tests, and technically stable during the experiment. We did not encounter major technical problems. Only one person in fifty one subjects stopped before completion of the visit. She was over-dressed and suffered from the heat.

Behavioral and physiological data.
During the visits the following data were recorded: -Proxemics, record of movements and direction of gaze of the avatar of the subject -Direction, distance and duration of the gaze, relative to the other avatars and to the works of art.
-Physiological data: heart rate, blood pressure, and cutaneous conductance.
-Behavioral data: trunk and head movements of the subject (Kinect).
-Video recording of the subject (See Figure  4a) and Video recording of screen view, third Person Point Of View. (See Figure 4b)

Selection of art works.
The members of the team selected arts works. The criteria for this selection were that they had produced in our members a strong emotional feeling. A collection of 120 arts works was established. Their emotional valence was then evaluated by an independent group of subjects in a on-line questionnaire enquiry with a subsequent statistical analysis.
Following this evaluation 2 groups of 48 Art works of similar characteristics were taken: the first one for hanging on the virtual exhibition walls and the second one for a subsequent questionnaire for memory (See below Protocole section).

Subjects.
It is known that gender differences exist for spatial memory and spatial orientation and therefore we chose to do the experiment only with female subjects. They were selected from students of ENSAD covered by the insurance of the school and as experience in their cursus. Their average age was 28 years. Their level of expertise in gaming, laterality, academic level, have been identified in order to group them for data analysis.

Protocol
Prior beginning the experiment, we explained to each subject that she was sitting on an art show jury. She had to select the works of art exhibited that appeal to her most. She was asked to carefully watch each work of art, in order to be able to vote, later on, for those for which she had the strongest feeling.
At the start of the visit, the subject was given some time to perform a few exercises in a training area. She then could familiarize herself with the device and take control of her avatar. This training time allowed us to validate that the subject was able to perform the visit, to move towards the works of art and to watch them. Then, the subject was invited to begin the visit.
Two groups of 48 art works each were selected from the original base of 120 Art works and hung randomly on the exhibition walls. The subject visited the rooms in sequence.
At the end of the visit, the subject was shown on a computer 96 images of artworks (a random combination of 48 seen in the environment, 48 new artworks). For each image, the subject was asked to decide if the displayed artwork has been seen in the environment or not. If yes, she has to answer questions to vote for and evaluate the works of art, in light of their experience, and to give us a feedback regarding their perception of the behaviour of the other avatars encountered all along their visit.

Conditions with respect to avatar presence.
During the visit of the different rooms, the avatar of the subject could be in three different situations: -Condition (solo) (S). The avatar of the subject was alone in the room.
-Condition "passive avatar" (PA). The avatar of the subject was in the presence of another avatar visiting the same room. A manipulator (see above) who could not be seen by the subject animated this avatar. This avatar did not seek interaction. It always watched another work than the one the avatar subject was observing. We shall call this avatar "passive" because it did not seek interactions -Condition (active avatar) (AA). The avatar of the subject was in the presence of another avatar visiting the same room. A manipulator (see above) who could not be seen by the subject animated this avatar. This avatar seeked interaction with the avatar of the subject. It always watched the same work than the one the avatar of the subject was observing.
Each of these conditions was presented in 4 successive rooms and randomly for each subject.

DATA ANALYSIS
In this chapter, we want to give a critical feedback on the methods used in this experiment. The values presented in this article are raw results. The relevance of its results must be evaluated by an Anova-type statistical analysis. (Analysis of variance). Their interpretation is in progress and will be the subject of another article. One of the objectives of this experiment was to test different techniques allowing us to approach the emotional state of the subject, by studying his behavior during his immersion in a paradigm. To our knowledge, there is no technique to measure directly this state. We chose indirect methods that were not constraining for the subjects: Proxemic measurements inform us about the activity of the avatar of the subject in the paradigm: recording the trajectories of displacement of the avatars, their orientation of point of view and the location of the paintings in the paradigm. We also recorded a video from the point of view of the subject in the paradigm, including his avatar (POW 3rd person). The biological and behavioral measurements of the subject in the cabin. We have recorded, heart rate, skin conductance, bust and head movements of the subject and a subject video during the visit. On the basis of these raw data, we have developed two study strategies: -A statistical approach, based on the data of all the subjects studied, according to the conditions S-AA-PA -A subject-based approach, based on viewing the recorded videos and analyzing the visit graph. These graphs gather all the recorded data, allowing us to identify and study specific behaviors.

Proxemic measures:
We have recorded the initial conditions of the paradigm, the geometry of the rooms, the respective positions of the art works exhibited randomly in the rooms, as well as the trajectories of displacement and orientations of the avatars in the paradigm. This data was reliable as it was directly recorded digitally via Unity. It was necessary to exclude certain periods of registration when subjects were hindered by external events during their visit. (eg dressing problems, intervention of the manipulator on a technical problem, etc.) Based on these data, we calculated proxemic values: the distance between the different avatars in the rooms, the distance of the subject avatar from the works, and we measured the viewing time of each work. These data were then studied according to the conditions S-AA-PA with the aim of detecting a possible influence of the copresence on the behavior of the subjects. (See Figure 5 and 6) We note that the time of presence in the rooms and the time of gaze on the art works are slightly higher when the avatar of the subject is in co-presence with another avatar, in comparison with the situations where the avatar of the subject is alone (see example above). At this stage of the study, the results cannot be formally interpreted as proving that the co-presence of an avatar has a direct influence on the behavior of the subject. The strategies of displacement prove very variable from one subject to another. It seems, in particular, that the level of experience at stake of the subject has a strong influence on the behavior of his avatar. It also seems necessary to study the influence of contextual elements, such as the emotional power of the works viewed.

Biological measurements:
These measurements were recorded on the subject during the immersion in the paradigm.

Heart rate
The measurement of the heart rate of subjects is recorded by a sensor placed on the left index finger of the subject. This measure is reliable and simple to implement.
We have seen some temporary breaks in recording during the visit, related to the hand movements of the subjects (loss of electrical contact). These breaks were detected and the measurements were excluded from the data. (See Figure 7) The mean heartbeat of the subjects during the visit is: 95.3 beats / minute. This value, higher than that of a subject at rest, suggests that he is involved in the task during the immersion. We found no significant heartbeat differences according to the conditions S-AA-PA. It would be interesting to study whether there is a variation in the heart rate, according to other parameters: -The distance between the avatar of the subject and the avatar Manip, when the latter is in the privacy zone of the avatar of the subject.
-The emotional level of the artwork being watched. (See Figure 8) We see a significant variation in heart rate depending on the experience level of the subject. "Confirmed" subjects averaged a heart rate greater than 20 beats / min, compared to subjects with no experience at play. This result suggests that subjects with strong experience in video games are involved more easily and more intensely in the task. The average heart rate seems to be an interesting indicator to measure the general involvement of the subject in the task. The analysis of changes in cardiac rhythm, not yet performed, could be an indicator of the emotional state, interesting to evaluate.

Trunk and head movements.
Movements detected within a pre-defined triangle, from shoulders to the subject's head, were measured using a Kinect during manipulation. The curves of these movements inform us about the rate of activity of the trunk and head of the subject. This measure was technically reliable. Particular attention must be paid to the proper functioning of the tracking during the start of the visit. The main difficulty encountered is to differentiate the movements directly related to the activity of the subject's avatar in the paradigm, from those due to real external disturbances that interfere with the subject in the cabin. We have identified the periods of overactivity of the subject, and we have determined the exact origin of it by viewing the videos. Since the origin of the activity was clearly related to external perturbations and not to the action within the paradigm, we excluded them from the data to be analyzed. We note a high variability in amplitude of torso motion between subjects, with a tendency to be more static for subjects with experience in game, as if they were more focused on manipulating the joystick and more immersed in their avatar. (See Figure  9) When the subject is in AA condition, we observe that the general activity of his bust is greater than those recorded when he is in AP condition and / or in S condition. We could interpret this difference as a sign of the influence of co-presence on the behavior of the subject. (See Figure 10) When we compare the movement of the bust when the subject looks at the works, the average activity of the busts is about 30% higher and we observe a reversal of the trend between the S PA condition and the AA with respect to its general behavior during visit. The activity of the bust and the head seems to be an indicator to be taken into account. It would be interesting to associate it with a measure of the orientations of the head, as well as measurements of the direction of viewing (eye-tracking).

Cutaneous conductance.
Changes in Cutaneous conductance are proved indicators for detecting changes in the subject's emotional state. These data were recorded using a sensor on the index and the middle finger of the subject's left hand. The amplitude and the quality of the signal obtained varies greatly according to the subjects. It is difficult to automatically detect curve variations for statistical analysis. Significant signal variations were detected visually. A verification by viewing the videos was done to identify their origins by checking whether they were related to an event affecting the avatar of the subject in the paradigm, or the subject in the cabin. (See Figure 11) The variations of cutaneous conductance were valuable indicators for the detection and the study of specific situations. (See Figure 12)

Case studies.
In this approach, the focus was on the behavior of a subject during his visit. we seek to identify specific strategies or situations that can inform us about the influence of co-presence on the subject's actions. We visually gathered all the data collected in the form of a temporal graph of the visit, which we synchronized with the video recording of the subject in the cabin, and the video record from the point of view of the subject on the paradigm. (See Figure 13) The graph shows x time (identical to that of the videos), and y the distance traveled by the subject during the visit. The S, PA and AA conditions are identified by a color code Green Red Blue. Dotted lines mark the passages from one room to another. The time of gaze on each artwork is materialized by a column, on which is stipulated the reference of the artwork. Depending on the viewing time, this column is more or less wide. On this basis are superimposed the cutaneous conductance curve (Skinvolt), the heart rate curve (Heartbeat), the trunk and head movements curve (HeadMovement), distance in avatars (Proxémy), and the presence of avatar Manip in the field (Avatar presence).
the crossing of these graphs and the viewing of recorded videos of each visit allowed us to identify and study specific behaviors.
By studying variations or correlations of signals, we were able to identify specific situations and verify their origin by referring to the videos of the subject concerned. Similarly, specific situations that we had identified by watching the videos could be analyzed more precisely by referring to the graphs. We have identified several types of special situations: -People trying to avoid being near the Avatar Manip -People testing the Avatar's reactions by pushing them with their avatar -Individuals with physical nervous attitudes in the cabin, or directly calling the Avatar Manip.
-People showing social embarrassment in the cabin.
As an example of a particular situation, we present an interesting case of a subject presenting discomfort in the cabin. (See Figure 14) In PA condition at timing (T 1539), the subject looks at the painting "The Origin of War" by Orlan (male version of "The Origin of the World" by Gustave Courbet). The subject looks at the picture with attention, without mentioning a particularly strong emotion, he smiles amused. Then the subject turns away from the painting and discovers that the Passive Avatar is present in the room to his left. He looks at another picture. The subject then shows surprise and embarrassment (T 1555). She turns her gaze away to look in the opposite direction to where the Passive Avatar is. Then she looks to the floor of the cabin, out of the screen, and releases the joystick to rub her hand. At (T 1559), we note a strong peak of skinconductance going from 0.52 to 0.83.. It would seem that it is not the painting itself that provokes this emotional peak, but the fact that the subject becomes aware that his avatar was in the presence of another avatar who could have observed it while he was watching the painting. A possible interpretation is that the subject would feel a social embarrassment even though the experience is experienced through his Avatar and that the subjects are physically distant. If this interpretation of this situation were to be validated by other similar observations, this would be a conclusive evidence that the co-presence of an Avatar can exert an emotional influence on the subject itself.

CONCLUTION
The data from this first research are still being analyzed, but these first results already allow us to consider new experiments. This experience convinced us of the richness and effectiveness of the arts and sciences research approaches, opening the dialogue between disciplines and bringing together practitioners and theorists on common problems, in an interdisciplinary approach. Another result of this research is the development, in a flexible and inexpensive technological environment, of a common experimental platform « Perception of shared spaces » which allow artists, in the same environment, to produce and distribute experimental and Scientists to observe the processes of creation and reception of these works. Together, they can understand their cognitive and perceptual phenomena and imagine future applications. We wish to continue this process, develop this tool by opening it to other disciplins: art history, evaluation of hybrid works (theater, music, online performance), organization and spatialization of information, Ars memoriae, teaching and new intuitive online forms to shared art and culture (libraries, media).