Effects of Interaction Level, Framerate, Field of View, 3D Content Feedback, Previous Experience on Subjective User eXperience and Objective Usability in Immersive Virtual Environment

This paper investigates the effects of interaction level, framerate, field of view, 3D content feedback and previous experience on subjective User eXperience (i.e., presence, engagement, immersion, flow, emotion, skill, experience consequence, judgement, technology adoption) and objective usability in immersive virtual environment. Data were collected from a series of five subexperiments (i.e., one for each influential factor) that involved a total of 152 individuals. The participants were asked to use the “Think and Shoot” immersive virtual environment and to complete a User eXperience questionnaire. Their subjective perceptions and objective measures were collected and analyzed. The results revealed that interactivity level and previous experience had an effect on subjective User eXperience and on objective usability. Framerate and field of view had an effect on objective usability. Finally, 3D content feedback had no significant influence on User eXperience. From these findings, key points for User eXperience practitioners are proposed. Index Terms Immersive Virtual Environment; Subjective user experience; Objective usability.


INTRODUCTION
User eXperience (UX) research in VR tries to understand how human experience the interaction within a Virtual Environment (VE). Plus, one goal of virtual world designers is to create virtual environments that provide greater UX. One way to reach this goal is to ensure designers assess UX in the early stages of the VE design process (Yogasara, 2014) and provide frameworks with appropriate guidelines that take into account all factors affecting the UX. Previous works from Lin  The model for VE from Lin & Parker (2007) focuses on influential factors specific to the system properties. This model introduces field-of-view, motion frequency, level of interactivity, visual interventions feedback as factors leading to an optimal UX (defined by a higher presence, a higher enjoyment and a lower simulator sickness). Presence is determined by engagement and immersion according to the authors. Presence is a component defined as the user's "sense of being there" in the VE. Engagement is a component defined as "a psychological state experienced because of focusing one's energy and attention on a coherent set of stimuli or meaningfully related activities and events" (Witmer & Singer, 1998, p. 227). Immersion is a component defined as the "objective level of sensory fidelity a VR system provides" (Bowman & McMahan, 2007, p. 38). Simulator sickness is an example of experience consequence which is a component we defined as the symptoms (e.g. simulator sickness, stress, dizziness, headache …) the user can experience in the VE. The authors revealed that a high level of presence in a VE, is significantly associated with a visual scene with large field-of-view (e.g., 180°), with very low motion frequency (0.03 Hz) and with an active mode of interaction. Furthermore, they added that a high degree of enjoyment (a determinant in the flow component) in a VE is also significantly associated with a visual scene with very low motion frequency, with an active mode of interaction and with visual interventions feedback to predict upcoming motion. The authors finally found that a visual scene with small field-of-view (e.g., 60°), with very low motion frequency, with an active mode of interaction and with visual interventions feedback to predict upcoming motion alleviate experience consequence (i.e., after effects) such as simulator sickness induced in a VE. The authors seem to point out the difficult challenge of providing at the same time a higher presence (i.e., with large field-of-view) and low cybersickness (i.e., with small field-of-view) in the VE.
The Component User Experience model (CUE-model) for human-technology interaction presented by Mahlke (2008) introduces system properties and user characteristics as categories of factors that mainly influence objective usability (i.e., task completion rates, time on task) and subjective UX seen as perception qualities, emotional reaction, experience consequence. Usability is a component defined as the ease of using (i.e., efficiency, effectiveness and satisfaction) the system. Emotion is a component defined as the subjective feelings (i.e., joy, pleasure, satisfaction, frustration, disappointment, anxiety …) of the user in the system. The author observed that system properties (e.g., image quality, interactivity, feedback, design) primarily impact objective usability but also quality perceptions (e.g., perceived usability, visual aesthetic, haptic quality), emotional reactions as well as consequences of the experience such as overall judgement (i.e., positive, indifferent or negative), decisions between alternatives technologies and their usage. User characteristics (i.e., age, aesthetic preferences due to previous experiences, cultural background, expectations and needs also shaped by previous experience) impact subjective feelings (e.g., emotion).
The virtual learning environment model from Shin et al. (2013) introduces a user characteristic. It was built from acceptance and continuance usage of technology theories. The technology acceptance is defined as the actions and decisions taken by the user for a future use of the VE. The authors found that users' previous learning experience with technologies is a factor influencing a specific UX component: intention to use the technology. Indeed, learners pass through distinct stages as they become confident in their use of the technology. Feelings of tension and frustration might more often impact users with less learning experience of the technology and affect the way they might want to adopt the technology (e.g., intention to use, purchase behavior).
The flow model for VE from Cheng et al. (2014) introduces three influential factors identified as interactivity, vividness and involvement that impact some components of the UX which are flow, telepresence and skill. The flow is defined as an "holistic sensation that people feel when they act with total involvement" (Csikszentmihalyi, 1975, p. 36). The skill is a component defined as the knowledge the user gain in mastering his activity in the virtual environment. The authors observed that interactivity level, a factor depending on media content, influences skill, flow and telepresence. Vividness (i.e., quality of information such as framerate), another factor depending on media content, influences telepresence. Finally, involvement, representing users experience (i.e., expert/novice distinction or importance given to the role), impacts skill and telepresence.
These previous studies give an overview of influential factors' impact on UX components. However, the focus on factors and UX components seems incomplete. They either focus on one or few factors (e.g., previous experience or interactivity, involvement and vividness), on a unique or a few UX components (e.g, intention or presence, enjoyment and simulator sickness) or they do not specifically focus on precise factors (i.e., system properties) and UX components (i.e., perception qualities). Therefore, the aim of the present study is to validate the impacts of 5 influential factors (i.e., interaction level, framerate, field of view, 3D content feedback, previous experience) on subjective UX (i.e., presence, engagement, immersion, flow, emotion, skill, experience consequence, judgement, technology adoption) and on objective usability (i.e., completion time, the number of errors, the levels score and the level reached).
In the remainder of this paper, first, we introduce the aim of our study and the four hypotheses resulting from the four previous studies in section 2. Second, we explain our methodological choices detailing participants, procedure, material and measures and we describe the data analysis in section 3. Third, we present the results of the impact of 5 influential factors on subjective UX components and on objective usability in section 4. Fourth, we discuss the validity of our findings and for each hypothesis we highlight the confirmed or unconfirmed previous impacts in section 5. Finally, we conclude with suggestions on how to use these findings to improve the subjective UX components and the objective usability in VEs.

II. AIM OF THE STUDY AND HYPOTHESES
Our study aims at characterizing the impacts of five influential factors (i.e., interaction level, framerate, field of view, 3D content feedback, previous experience) on subjective UX components and objective usability for IVEs. User's previous experience with 3D technologies were measured through two self-efficacy questions on the use frequency of 3D technologies such as 3D interaction devices (i.e., headset, gamepad, …) or 3D video games and 3D software (i.e., Unity, 3DSMax, …). Four hypotheses were proposed to validate these five influential factors. Lin & Parker also noted that enjoyment (i.e. flow), presence (i.e., immersion and engagement) and experience consequence are associated with motion frequency of the visual scene. Cheng et al. suggest that vividness (i.e., the quality of information displayed such as framerate) enhances presence. Lin & Parker again revealed that a higher presence is significantly associated with a larger field of view (e.g., 180°) whereas a lower experience consequence (i.e., after effects) is significantly associated with a smaller field of view (e.g., 60°). They also noted that enjoyment and experience consequence are associated with visual intervention feedbacks. Mahlke (2008) present system properties such as presentation (e.g., field of view, framerate), functionality (e.g., level of interactivity) or dialogue (e.g., feedback) as determinants of perceived qualities, emotions, user's judgement, choices and usage behavior toward the technology.

Impact of Five Influential Factors on
Objective Usability. Mahlke (2008) suggest that system properties such as presentation (e.g., field of view, framerate), functionality (e.g., interactivity level) or dialogue (e.g., feedback) are factors that primarily impact objective usability (i.e., task completion rates, time on task). Hence, we may hypothesize that higher values of field of view, framerate, 3D content feedback and interactivity level positively impact the objective usability (Hypothesis 3). Mahlke also noted that the influence of user characteristics, such as aesthetic preferences, expectations or needs shaped by previous experiences, on objective usability have been assumed from previous studies (Plocher, et al., 1999). Consequently, we may assume that previous experience with 3D technologies positively impacts the objective usability (Hypothesis 4).

Participants.
152 participants (28 women and 124 men) who had volunteered to take part in this study were randomly assigned to five subexperiments within which we control the five factors independently (see Table 1). Each participant was assigned once to one of the five sub-experiments (i.e., participants in sub-experiment n°1 were all different from participants in sub-experiment n°2 and in sub-experiment n°3, etc). Participants were asked their last diploma or their current occupation. 122 participants worked or studied in Information and Communication Technology (ICT) or Computer Science fields (e.g. VR, network, web, graphic design, multimedia and internet, …). The other participants worked or studied in various other fields (e.g., education, marketing, food service, public relations, bank, …). The sample mean age was 23.96 years and the standard deviation was 6.93 (ranging from 18 to 63). In the fifth subexperiment, the sub-experimental groups were significantly different in terms of previous experience in 3D technologies (previous experience group: M = 15.09, SD = 5.36; no previous experience group: M = 6.46, SD = 2.81; t(70) = 8.62, p < 0 .001, twotailed). The first four sub-experiments were designed with a paired sample for two reasons: first, it allowed us to reduce the number of participants that we would had difficulties to find, second, this configuration is the best way to control that the dependent groups had the exact same previous 3D experiences. In these four sub-experiments, the passing order was counterbalanced (i.e., in sub-experiment n°1, the first participant tested FOV of 32° and then 106°, the second participant tested FOV 106° and then 32°, the third participant tested in the exact same order than the first participant, etc).

Procedure.
The participant began with filling an identification survey. Then, the experimenter introduced and gave the participant instructions about the experiment. He could finally sit on a typical office chair and run through three sessions for sub-experiments 1 to 4, and two sessions for sub-experiment 5 in the VE (given the independent experiment design). The first one was an orally guided training session of about 5 minutes for all the sub-experiments. The participants could ask for more or less training time if they felt more or less comfortable in the IVE. The second and third sessions for sub-experiment 1 to 4 and the second session for sub-experiment 5 were regular sessions of 5 minutes. For these sessions, participants followed the instructions in a pseudo programming language written on a panel in the application. The tasks covered different aspect of "hunting" activity such as moving, shooting, searching around. A questionnaire on UX completed the session (excluding the training session). Each participant spent between 45 minutes to 1 hour for subexperiments 1 to 4, and between 30 to 45 minutes for sub-experiment 5.

Material and Measures.
Virtual Environment. The edutainment VE "Think and Shoot" was designed with the development tool UNITY © and ran on a Dell 64bits computer with 4GB of RAM, an Intel® Xeon® processor, CPU E5-1603 2.80GHz. A Logitech wireless gamepad, a Dell keyboard and an Oculus development kit 2 (DK2) allowed the participant to collect balls and to shoot on three different sphere targets in the training session and on two different evil creatures in the regular sessions according the instructions given on a panel in the application (Figure 1). 3D spatialized sound was rendered in a Tritton AX 180 audio headset. The VE consists of two main actions which are collecting balls and shoot on evil creatures. The participants are given instructions in a pseudo programming language where the action of shooting is represented by a function and the three parameters of the function are the type of evil creature to shoot on (i.e., two distinct types of creatures), the type of ball to shoot (i.e., three distinct types of balls) and the remaining balls that can be used. An example of instruction could be "Shoot (fire creature, ice ball, 0)", the participants should understand that there is no ice ball left to shoot on the fire creature, they must collect more. The aim of this VE is to familiarize the participants with the notions of function and parameters during the 6 levels of the VE. The levels gradually increased in difficulty (i.e., more creatures to eliminate, and more categories of balls). The participants gained a point for each eliminated creature, they lost a point at each collision of their avatar with a creature. The avatar "dies" when there are no more points and the VE re-launches at the beginning of the same level. The interactivity level (rotation or no rotation), the 3D content feedback (minimap or no minimap), the framerate (i.e., 30 or 70 FPS) and field of view (i.e., 32° or 106°) values were configured in the VE for the dedicated subexperiments. Significant values were chosen for the influential factors. The value 32° was chosen as the minimum acceptable value for the user to be able to interact with the VE (Hassan et al., 2007) and the 106° value was chosen as the widest field of view in the DK2 and thus the most comfortable. Then, the 30 FPS value was chosen as the limit (but still comfortable) for users to be able to perceive framerate change (Claypool & Claypool, 2007) and the 70 FPS value was chosen to avoid as often as possible latency (Oculus Best Practices, s.d.). The minimap was chosen as a navigational feedback of the VE to help the user acquire better spatial knowledge from the combination of two types of navigations (i.e., map and VE) (Richardson et al., 1999). Finally, the body rotation is part of the basic interaction in VEs. It contributes to the proprioceptive feedback of body movement perceived by the user (Slater & Wilbur, 1997). Identification Survey. Two matrix scale questions were dedicated to 3D technology expertise (0 = Never, 1 = Little, 2 = Sometimes, 3 = Often, 4 = Always). The first matrix scale question was dedicated to the usage frequency of interaction devices such as VR headset, gamepad, joystick, Kinect, leap motion… and the second matrix scale question was dedicated to usage frequency of 3D video games and 3D software (i.e. Virtools, Unity, 3DSmax, Maya, AutoCAD, Architectural desktop …). UX Questionnaire. Our UX in IVE questionnaire of 68 items is used to assess the subjective UX (see appendix 1). All items and questions were originally in French. The subscales items were provided from nine original questionnaires (i.e., PQ, ITQ, Flow4D16, CSE, AEQ, UTAUT, AttrakDiff, SSQ). It comprised nine subscales and composed of 68 items and 3 open questions. 9 items compose the presence subscale; 3 items compose the engagement subscale; 5 items compose the immersion subscale; 10 items compose the flow subscale; 11 items compose the emotion subscale; 6 items compose the skill subscale; 9 items compose the judgement subscale; 8 items compose the experience consequence subscale; 7 items compose the technology adoption subscale. The participants' UX scores were collected through a 10-point Likert scale (1 = strongly disagree, 10 = strongly agree) or a semantic differential scale: point 1 was coded as a negative-connoted adjective (e.g. impractical, confusing, amateurish …) whereas point 10 was coded as a positiveconnoted adjective (e.g. practical, clear, professional …). The questionnaire was validated through reliability (i.e., subscale internal consistency and items consistency) and sensitivity analyses. Subscales internal consistency showed Cronbach's Alpha values ranging from 0.398 to 0.908 (Table 2) (Tcha-Tokey et al., 2016). Usability Measures. Objective usability was measured through level completion time, number of errors, total levels score and level reached. Level completion time is the average time the users took to complete the level n°2 (i.e., level reached by all participants). The number of errors is the average number of errors (i.e., number of lost points when the user is touched by a creature) during the whole session. The total levels score is the average total score gained in all levels, and the level reached is the average level the users reached at the end of the session.

Data Analysis. 3.4.1 Effects of 5 Influential Factors on
Subjective UX. The UX questionnaires from the sub-experiments were analyzed. The Test of Normality (i.e., the Shapiro Wilk) indicated a non-significant difference (p > 0.05) suggesting a validation of the assumption of normality. So, a pairedsamples t-test was used to analyze scale scores of the questionnaires from subexperiments 1 to 4, and an independent sample t-test was used to analyze scale scores of the questionnaires from sub-experiment 5.

Effects of 5 Influential Factors on
Objective Usability. Objective usability was analyzed. A paired-samples t-test was used to analyze the average level completion time, the number of errors, the levels score and the level reached in the VE from subexperiments 1 to 4, and an independent sample t-test was used to analyze the same measures from sub-experiment 5.

IV. RESULTS
The results of paired and independent sample statistics are detailed below. They show the effects of 5 influential factors on subjective UX and objective usability.

Effects of Field of View, Framerate, Interactivity Level and 3D Content Feedback on Subjective UX.
Field of view (FOV), framerate (FR), interactivity level (IL) and 3D content feedback (3DCF) should affect the user's subjective UX (i.e. presence, immersion, engagement, flow, skill, emotion, experience consequence, judgement and technology adoption). To check this assumption, we conducted paired sample statistics. They revealed that IL actually impacts presence, engagement, flow, skill, emotion, judgement and technology adoption (see Table 3). However, FOV, FR and 3DCF show no impact on any of the subjective UX components. Indeed, in the gamepad (IL) condition, the presence is higher (t = 6.20, p < 0.0001), the engagement is higher (t = 4.96, p = 0.0001), the flow is higher (t = 3.48, p = 0.003), the skill is higher (t = 5.81, p < 0.0001), the emotion is higher (t = 3.57, p = 0.002), the judgement is higher (t = 2.90, p = 0.01) and the technology adoption is higher, (t = 3.07, p = 0.006).

Effects of Previous Experience on Subjective UX.
Previous Experience (PE) should affect the user's subjective UX (i.e. presence, immersion, engagement, flow, skill, emotion, experience consequence, judgement and technology adoption). To check this assumption, we conducted independent sample statistics. They revealed that PE actually impacts engagement, skill, and technology adoption (see Table 4).

4.3
Effects of Field of View, Framerate, Interactivity Level and 3D Content Feedback on Objective Usability. Field of view (FOV), framerate (FR), interactivity level (IL) and 3D content feedback (3DCF) should affect the user's objective usability. To check this assumption, we conducted paired sample statistics. The data rather reveal that FOV, FR and IL have an impact on some of the objective usability metrics (see Table 5). However, 3DCF has no impact on any of the objective usability metrics. Indeed, level reached is higher in the 106° (FOV) condition (t = 2.13, p = 0.046), in the 70FPS (FR) condition (t = 2.36, p = 0.03) and in the gamepad (IL) condition (t = 3.39, p = 0.003). The total score is higher in the 70FPS (FR) condition (t = 2.91, p = 0.009) and in the gamepad (IL) condition (t = 3.96, p < 0.001).

Effects of Previous Experience on Objective Usability.
Previous Experience (PE) should affect the user's objective usability. To check this assumption, we conducted independent sample statistics. The data indeed reveal that PE have an impact on objective usability (see Table 6). In the previous 3D experience (PE) condition, the level reached is higher (t = 3.43, p = 0.001), the total score is higher (t = 3.6, p < 0.001), the completion time is lower (t = 2.1, p < 0.05) and the number of errors is lower (t = 2.37, p < 0.05).

V.
DISCUSSION This present research investigates relationships between influential factors specific to the system properties (i.e., field of view, framerate, 3D content feedback and interactivity level) and influential factors from user characteristics (i.e., previous experience) on subjective UX (i.e., presence, immersion, engagement, flow, skill, emotion, experience consequence, judgement and technology adoption) and objective usability (i.e., level reached, total score, completion time, number of errors). Our first hypothesis stating that higher values of field of view, framerate, 3D content feedback and interactivity level positively impact the subjective UX is partially validated. The results rather show that interactivity level impacts the major part of UX components. Our result first shows that interactivity level impacts presence. This finding is consistent with other studies demonstrating the impact of interactivity on presence (i.e., Welch et al.   Shin et al., 2013). Our third hypothesis stating that higher values of field of view, framerate, 3D content feedback and interactivity level positively impact the objective usability is partially validated. The results rather revealed that level reached is impacted by interactivity level, framerate and field of view; total score is impacted by interactivity level and framerate. The impact of interactivity level on usability is related by to the well-known Fitts' law (i.e., Fitts & Peterson, 1964;Cabral et al., 2005). The effect of framerate on objective usability has also been noted by Richard et al. (1996): they specifically investigated the effect of framerate on the time required to grasp a moving target (i.e., a virtual object) in a VE. The impact of field of view on objective usability has also been noted by Polys

VI.
CONCLUSION This paper questions relationships between influential factors, subjective UX and objective usability. First, we characterized the impact of influential factors (i.e., interaction level, framerate, field of view, 3D content feedback and previous experience) on subjective UX (i.e., presence, engagement, immersion, flow, emotion, skill, experience consequence, judgement, technology adoption). Second, we analyzed the impact of influential factors on objective usability (i.e., level reached, total score, completion time, number of errors). The results show that interactivity level has an effect on all subjective UX components except immersion and experience consequence and on 2 usability metrics. Indeed, the presence, the engagement, the flow, the skill, the emotion, the judgement, the technology adoption, the level reached and the total score are significantly higher in the gamepad condition than in the keyboard condition. Previous experience has an effect on 3 subjective UX components and on all objective usability metrics. Indeed, the engagement, the skill, the technology adoption, the level reached, the total score are significantly higher in the previous 3D experience condition than in the no previous 3D experience condition whereas the completion time and the number of errors are lower in the previous 3D experience condition than in the no previous 3D experience condition. Framerate has an effect on 2 usability metrics. Indeed, the level reached and the total score are higher in the 70FPS condition than in the 30FPS condition. The field of view has an effect on a unique usability metric. Indeed, the level reached is higher in the 106° condition than in the 32° condition. These results suggest some recommendations to help designers focus on the unsatisfactory aspects of the VE in terms of subjective UX and objective usability. Indeed, we can raise key points for practitioner's: • Use interaction technologies such as gamepads with high interactivity level to enhance the subjective UX (i.e., presence, engagement, flow, emotion, skill, judgement and technology adoption) and objective usability (i.e., higher level reached and higher total score). • Provide specific VE design according to the user's previous experience with 3D technologies (i.e., expert/novice, importance given their role) to increase his degree of engagement, his perceived skill, his propensity to adopt the VE and his objective usability (i.e., higher level reached, higher total score, faster completion time, less errors). • Use high framerate (e.g., 70FPS) to increase objective usability (i.e., higher level reached and higher total score).
• Use large field of view (e.g., 106°) to increase objective usability (i.e., higher level reached). Future perspective of this work would be dedicated to understanding more deeply how independent variables (i.e., interactivity level, previous experience, …) impact dependent variables (i.e., subjective UX and objective usability).

VII.
ACKNOWLEDGEMENTS Our thanks go to Laval Agglomération and the Mayenne department for their financial support, and to the participants who took time to engage and provide feedback in this project.

APPENDIX 1.
English translation of our v2 unified UX in IVE questionnaire (originally in French).