Comparing Conventional versus Immersive Service Prototypes: An Empirical Study
Abstract
For years, Immersive Technologies and 3D printing, demonstrated their capacity to quickly build product prototypes in order to reach a common understanding among all stakeholders, especially potential users. Service prototyping is a novel agile process intended to accelerate the service development, while improving the overall anticipated service experience. The use of Immersive Technologies in service prototyping is intended to enable a co-creative and explorative service experience, even before the service really exists. Service prototyping transforms intangible processes into a real experience. Immersive Technologies are already deployed in several industrial applications ranging from product design to product and service exploration. They are also used for conducting training even before the product or service exists. The main concern remains in the fact that there is a lack of study for comparing and selecting the most appropriate form of Service Prototypes (SP) to explore a new service. This paper presents our empirical study comparing different SP forms and the results of two experiment sessions that were conducted at ENSAM Laval and Angers campuses. These results reveal that participants preferred immersive forms rather than conventional forms. However, it also unveils some difficulties in properly handling Immersive Technologies.
Introduction
Service is defined as a process, in which organizations are intended to deliver intangible and tangible features, that creates a value for the customer (ISO/IEC 20000-1 Information Technology, 2005). Because of the increasing complexity of products and services as well as their intricacy (Ostrom et al., 2015), immersive technologies, such as: Virtual Reality (VR), Augmented Reality (AR) or Mixed Reality (MR), are frequently used in the industry for creating product prototypes; it is a way for all stakeholders to explore and experience a product before it really exists (Buchenau and Suri, 2000). As a logical propagation, there is a current trend to use also those immersive technologies for prototyping services (Sämann et al., 2016). Innovative digitalized services and solutions play a significant role in defining the strategy of industrial companies (Ostrom et al., 2010). It is well accepted that service engineering traditional linear development methods necessitate a longer period of time compared to recent agile methods. Service prototyping is nowadays widely considered as an innovative service engineering process for optimizing a service and its development (Abdel Razek et al., 2017) according to users’ needs and expectations.
Service prototyping focuses on stakeholders’ co-creative experience, service process learnability and iterative service development. It is also an agile iterative process, which enables co-creative activities, engaging all service stakeholders at the early stage (Abdel Razek et al., 2017; Sämann et al., 2016). SPs have different forms. Conventional Service Prototype forms (CSP), here conventional means widely used and well known forms, like “Paper Service Prototype” (PSP), Verbal Service Prototype (VSP) and Mock-up Service Prototype (MSP). There are also technology-based forms named Immersive Service Prototype (ISP), such as Virtual Reality Service Prototype (VRSP), Augmented Reality SP (ARSP) and Mixed Reality SP (MRSP). CSPs are more popular because they are easier to deploy and do not necessitate specific equipment training and investment in technology, such as Head-Mounted Display (HMD). Whatever services need to be developed or re-developed, its iterative development process is getting more complex (Ostrom et al., 2015), which is also due to the digitization and new value creation approach (Ostrom et al., 2010). Consequently, SP attempts to transform an intangible service idea into a real experience allowing the service exploration, communication, and evaluation. This enables service stakeholders to undertake informed decisions about service alternatives or market introduction even at the early stage; these informed decisions are based on stakeholders’ feedback after using a service prototype.
This paper presents the results in terms of SPs performances; it is intended to help service stakeholders to select the most appropriate SP form. Several SP experiment sessions were conducted in France and Germany, along 2018, within an academic context. The goal is to compare the performance of Conventional Service Prototypes (CSP) versus Immersive Service Prototypes (ISP). This experiment is based on the use of a specific service scenario where participants have to disassemble and then reassemble a simple mechanical element composed of several parts. For completing these two tasks, participants have to use paper SP and video SP (conventional SP forms), as well as VR SP and AR SP (immersive SP forms) in different sequences. Overall, this empirical study, having more than 100 participants, is intended to statistically validate the research model and instrument presented in our previous publications (Abdel Razek et al., 2017; Abdel Razek et al., 2018a; Abdel Razek et al.., 2018b).
This paper consists of 5 sections: starting with this introduction, then, briefly presenting the literature review on existing service prototyping theories and concepts as well as our previous work in this research area. The “research methods” section discusses the motivation to apply mixed methods while describing the experiment. The “findings” section includes the analysis of collected data. Finally, we present the limitations of this study, some initial conclusion and future work.
Literature Review and Previous Work
Publication Trend
The carried out literature review unveils that numerous publications address prototyping and immersive technologies (VR, AR, MR) for products, systems or software (see Fig. 1). Comparatively, there is a pretty low amount of papers on service prototype and VR or AR or MR as shown in Figure 2. According to another Google scholar search, it appears that the combined term “Immersive Service Prototyping” or “Immersive Service Prototype” is brand new.
Figure 1: Literature Progression on Prototyping and VR or AR or MR 2007 – 2018
Figure 1 shows a current trend of an increasing publication amount that includes “prototyping” and “MR” (Mixed Reality), while the amount of papers including “prototyping” and “VR” (Virtual Reality) or “AR” (Augmented Reality) decline a little during the last year period. In the same way, Fig. 2 presents the current trend of publication amounts over 2007 to 2018 for VR SP, AR SP and MR SP. The formula used in google scholar was mainly “VR” OR “AR” OR “MR” AND “Service Prototyping” AND “Service Prototype”. “Immersive Service Prototyping” search came with almost zero results (only our published papers were found).
Figure 2: Literature Progression on Service Prototype (SP) and VR or AR or MR 2007 – 2018
Brief Literature Review
Service prototyping is intended to make service intangible aspects becoming more tangible (Balzert, 1989). It enables a co-creative cross-disciplinary service development process, in which all stakeholders take part in the service development process (Blomkvist and Holmlid, 2011). During the literature review, we identified nine papers reporting some forms of SP experiment as presented in the table 1. These reported experiments applied predominantly qualitative methods. They were mostly based on a case study within a real or simulated service scenario. Most of these selected existing papers experimented a service for which they used SP for designing and verifying the concept of this service. SP was intended to orient stakeholders within complex service scenarios happening in diverse industrial sectors. ISP was also intended to evaluate human behavior through user experience, and simulated service environments.
The mentioned papers in the table 2 mostly discussed the optimization of: (1) service processes (Exner et al., 2014; Arvola et al., 2012; Fukuhara et al., 2014); (2) the improvement of service designs (Kwon et al., 2015; Peng et al., 2017); (3) service training processes (Jung Bae and Seong Leem, 2014; Boletsis et al., 2017); (4) service prototyping methods (Blomkvist and Holmlid, 2012; Kuure et al., 2014). We identified one case that applied quantitative method to evaluate the appropriateness of a 3D VR service App that engaged 30 participants (Peng et al., 2017). In our previous comparative study (Abdel Razek et al., 2018a), we discussed the seven SP forms distributed into conventional and immersive service prototyping. The most widely used SP forms were the Paper SP (PSP), Mock-up SP (MSP), Virtual Reality SP (VRSP), and Augmented Reality SP (ARSP), these could be divided into two categories, Conventional SP (CSP) and Immersive SP (ISP).
Scholars have already stated that there is a lack of empirical studies on service prototyping (Exner et al., 2014; Blomkvist and Holmlid, 2012). This matches with the literature review we have done on service prototyping publications, which confirms a lack of empirical investigations. This lack of previous empirical investigations on service prototyping was in fact the motivation to carry out this study on comparing CSPs and ISPs.
Table 1: SP Experiments identified in the Literature Review
Case |
Research Method |
Ref. |
|
|---|---|---|---|
Quantitative |
Qualitative |
||
s-Scape Service Prototyping |
Case Study (The Theater, Hybrid, VR) |
(Jung Bae and Seong Leem, 2014) |
|
Virtual Body-storming Service Prototyping |
Role Play, Expert Evaluation Cognitive Walkthrough |
||
Service Prototyping Practice |
6 Expert Interviews |
||
SINCO Service Prototyping Methods |
Interviews, Use Case Data |
||
Evaluating Servicescape Designs |
Case Study (Duty-free Shop) |
||
3D Multiple Medical Imaging System |
30 Volunteers |
||
PSS Lifecycle Testing |
Case Study (Smart Hybrid Prototyping) |
||
Service Walkthrough |
Case Study (Augmented Tourism Services) |
||
Improving Service Processes Based on Visualization of Human-Behavior |
Case Study (Hospitality POS Data Visualization) |
||
(L: Low; M: Medium; H: High; V: Very)
Our Previous Work
Service Prototyping is a process for service development that uses service representations, or parts of it, in order to explore a service idea, communicate a concept and evaluate a design (Abdel Razek et al., 2017). SP covers different intangible and tangible aspects, different activities, and diverse stakeholders (Abdel Razek et al., 2017). In the past three years we have explored different SP forms, applications, and tools. Hence, we conducted several industrial surveys, numerous individual and focus group interviews, as well as workshops dedicated to service prototyping.
In our previous publications, we presented the inventive application of service prototyping in service innovation (Abdel Razek et al., 2017). The latest publication discusses the proposed service prototyping framework and model that were used to design this experiment (Abdel Razek et al., 2018b). One of our previous papers (Abdel Razek et al., 2018b), reports the experience feedback of industrialists that participated to a specific workshop dedicated to explore service prototyping and service prototypes. The value average for each SP and corresponding property is presented in the table 1. SPs are the outcomes of a service prototyping process; each of these forms will have an individual property rating to help with the selection process.
Table 2: Service Prototype Properties (based on (Abdel Razek et al., 2018a))
SP Type |
Fidelity |
Resolution |
Effort |
Interactivity |
UX |
|---|---|---|---|---|---|
PSP |
L |
L |
L |
L |
L |
MSP |
M |
M. |
M |
M |
M |
VRSP |
H |
V H |
V H |
V H |
V H |
ARSP |
V H |
H |
H |
H |
V H |
Finally, the way to find out the most effective and efficient SP form to implement for a specific purpose and service activity is rarely addressed in the literature (Abdel Razek et al., 2018). It results in a lack of understanding that hinders service designers to apply properly immersive technologies for designing more efficient and effective service prototypes. This study aims to give a clearer picture on the optimal application of these SP conventional or immersive forms.
Research Methods
Context
Overall, our SP experiment was conducted in three different locations: Furtwangen, Laval and Angers The main goal is to reach different academic target groups in order to achieve a higher level of participants’ diversity: (1) IT & Engineering (Furtwangen); (2) Software Engineering (Laval); and (3) Mechanical Engineering (Angers). This paper discusses the results of the two experiment sessions that were conducted at the Laval Virtual Center in Laval and at the Engineering School of ENSAM in Angers during October 2018.
The validation experiment has been designed to engage participants perceptually, emotionally and cognitively. Paper-based instruction leaflets are quite common in the industry for servicing machinery or even for more personal uses like assembling Ikea furniture. This kind of drawing-based instructions can be considered as a part of the service process. The instruction leaflet presents to the participants, for example, the necessary steps and elements for assembling or disassembling specific furniture. Therefore, for designing our experiment, we have imagined a service to support people needed to assemble or disassemble a mechanical element constituted of several parts in the most comfortable, efficient and reliable way. In this case, service stakeholders would explore, along the service prototyping process, different service alternatives through the use of different forms (PSP, MSP, VRSP, ARSP) of service prototype that users (the participants) will experience and provide feedback on the above-mentioned three aspects (comfort, efficiency and reliability).
The experiment engages the participants with two CSPs where participants read a leaflet or watch a video and two ISPs where participants interact with objects within a virtual reality environment or in an augmented reality environment. All participants utilize these SPs to complete the disassembling and assembling tasks by using the support of each of the different SP forms. The mechanical element to disassemble/assemble is composed of a three-part metal construction with coinciding components. The three-part metal construction has multiple screws and bolts in multiple locations, which needs to be unscrewed/screwed.
These physical tasks are guided by the different instruction forms: (1) VRSP for immersing the user through an HTC Vive HMD to explore and interact with the instructions in a virtual environment; (2) ARSP uses a tablet device to overlay the instructions of each disassembling and assembling step directly on the physical metal construction; (3) PSP includes all the necessary steps and explanation of every step through the CAD drawing of each part of the three-part metal object; (4) MSP contains a replication video of the assembling and disassembling processes to show to the user how to do it correctly.
Protocol
Every participant had to experience each service prototype form and provided their feedback in filling out a bipolar survey. The overall duration of a participant’s session was about one hour to complete the two tasks four times using the four different forms of service prototype and filling out the survey as well. During the experiment one researcher observed the participant’s actions and reactions; even when they were immersed in VR, observation on what they were doing was done through a monitor. Every participant’s interactions on using the ARSP was done through the tablet and the AR App. Physical observation was done also while they were using PSP and MSP and see if they needed any support. The duration of each task was recorded as well as noting their attitudes, comments, explanation requests, and errors.
Figure 3 illustrates the experiment protocol where every participant: (1) receives a verbal briefing and a written briefing; (2) fills out demographic questionnaire; (3) proceeds with their experimentation part according to their experiment cycle, (4) answers the bipolar survey after each experiment task, (5) contributes to the experiment debriefing.
The participants had to fill out four bipolar surveys throughout each individual experiment session. Each bipolar question presents a semantic scale with rating values ranging from -2 to 2. The questionnaire starts with the selection of the SP form that was completed. A table of the English survey can be seen in the Appendix. The tasks were clocked from the moment that participant starts the disassembly or assembly task till the end of each of these tasks. The participants were all assigned different SP cycles, as to eliminate biasing from doing the task with the same SP form at the start or end of a cycle. The experiment cycle consists of the combination of the SP forms used in a specific order (see Table 3). The SP cycle that the participant starts with changes every set of participants to eliminate biasing as well.
Figure 4: VR Lab at the Laval Virtual Center, Laval
The experiment in Laval was conducted at the Laval Virtual Center VR lab of the Presence and Innovation Team as shown in Fig. 4 and in Angers at the VR Lab at École Nationale Supérieure des Arts et Métiers (ENSAM) in Angers as shown (see Figure 5).
Both labs were well equipped with state-of-the-art VR systems, HTC Vive at VR Lab in Laval and HTC Vive Pro at VR Lab in Angers. Research colleagues, students and also faculty members have participated to experiment sessions.
Methods
As shown from the previous literature that mixed methods is an appropriate choice to tackle such an explorative study that has multiple facets (Creswell et al., 2003; Johnson and Onwuegbuzie, 2004). The mixed methods approach was already demonstrated as a proper method to investigate similar immersive experience studies (Krawczyk et al., 2017). The experiment collects each participant’s feedback by using a bipolar survey with a rating justification part for the qualitative data. There is also the clocking of the participant’s disassembling and assembling durations, the observed participant’s attitude during the task completion and the number of errors and explanation requests each participant had. Mixed methods approach combines quantitative and qualitative results on the use of CSPs and ISPs. The quantitative data highlights the subjective (perceived) and objective (measured) impacts of using immersive technologies on the service prototypes. The qualitative data explain the reasons or motivations behind the observed impacts and allow exploring in depth the resulting experience.
Mixed methods (Creswell et al., 2003) technique attempts to the interpretation of the findings by combining qualitative and quantitative data; using the qualitative data to explore and interpret the quantitative findings throughout the participants’ collected feedback. In this study a combination of quantitative and qualitative research is used because ISP is a novel service development process that is not extensively studied. The quantitative and qualitative data are collected and analyzed at the same time; the observations are elaborated on both the quantitative and qualitative results.
The bipolar questionnaire consists of 15 quantitative rating questions; each rating question includes a qualitative justification element in order to better understand the motivation explaining the level of submitted rating. These justification elements embedded in the bipolar questionnaire allow participants to freely express their feedback about their rating. This questionnaire was written firstly in English and then was translated into German (used at The Furtwangen University) and into French (used at ENSAM Laval and Angers Campuses). All questions were carefully selected to cover the SP experience facets, SP efficiency facets, and the real world disassociation factors as discussed in previous publication (Abdel Razek et al., 2018).
The Service Prototype eXperience (SPX) stems from the participants’ rating and feedback during the experiment. The SPX is reflected from the rating questions that shows if participants were convinced of that SP form; if they are willing to re-use the SP form in another context, and if they would recommend this SP form to others; the SPX is an extension on the Immersive eXperience model from Pallot (Pallot et al., 2017) built on the baseline of the holistic view of UX (Pallot and Pawar, 2012).
The real world disassociation factors are also extracted from the responses of the participants on the questions that revolve around the feeling of the notion of time disappearing during the experiment; the ability to be attentive to outside factors, and the responsiveness to external influences as it was previously used by (Pallot et al., 2017). The service prototyping efficiency is then measured by the timing of disassembly and assembly tasks for each SP form as well as the number of explanations asked and errors made by the participants.
Findings
The collected data were clustered to several data sets; the first set of results is the demographic data of the participants; the second set of results addresses the answered bipolar survey; finally, the third results set reflects the observation metrics.
Demographic Results
The demographic survey consisted of 10 multiple choice questions as follow: (0) the participant’s experiment ID and cycle sequence number of their experiment, (1) gender, (2) age slice, (3) occupation, (4) Service Design knowledge, (5) Immersive Technologies knowledge, (6) SP knowledge, (7) previous SP experience level, (8) previous ISP experience level, and (9) location of the experiment.
Figure 6: Experiment Participants’ Gender & Age Slice
A total of 48 participants volunteered to join in the experiment, 24 at LVC in Laval and 24 at ENSAM in Angers. The data shows that male was the predominant gender with 71% of the participants being males and only 29% were females. The age slices of the participants were also concentrated in the slice from 17 to 25 years with 74% of the participants; 12% for the age slice from 26 to 35; and 8% for the age slice from 46 to 65. These results are presented graphically in Fig. 6. The amount of young participants is representative of the academic setting where the experiment sessions took place. The unbalanced gender can be deduced from the fact that the experiment took place in the engineering schools in Laval and Angers where the majority of the students are male; this is reflected in the results were only 29% of participants are females.
The participants were primarily students (engineers, master’s degree) ranking up to 79% of the total participants, followed by 7% professors, and then 6% PhDs. This was expected as the experiment sessions were conducted in academic environments, which the majority had no prior service prototyping experience, which is quite important to capture their first impression of such a novel process. The occupation breakdown chart appears in Figure 7.
The diversity of the participants is reflected on their current knowledge levels, especially in the area of immersive technologies. Fig. 8 highlights that 46% are at beginner level, 25% belong to practitioner level, 19% declared themselves as specialists, 8% had no prior knowledge, while only 2% where experts. Furthermore, 50% of the participants had no knowledge of service prototyping while 44% were at beginner level, and 6% were practitioners.
This is done for assessing the level of prior theoretical knowledge in both service prototyping and immersive technologies (VR, AR, MR), which determines the significance of that theoretical knowledge influence on the participants. These indications could be very beneficial to observe the short and long learning process effects of using immersive technologies and service prototyping as well.
Another worthwhile aspect to quantify is the efficiency of the participants during the experiment, which is related to knowledge of the participants reflected in the prior practical experiences in both CSP and ISP. As shown in Fig. 9, 77% of the participants had no prior service prototyping experience, while 17% had experienced CSP, 4% had experienced ISP, and only 2% had experienced both.
The results of the knowledge levels are understandable as 79% of the participants were students, where the knowledge levels are still at the starting point or beginner level at best. This is important as it reflects the opinions of students, the engineers and service designers of the future. These demographic data are representative of the different technical disciplines in both Laval and Angers. The goal was to have the widest possible spectrum of participants in an academic environment, while having new and fresh feedback from novice participants.
Bipolar Survey Results
The questionnaire of the bipolar survey was devised and modeled in order to cover 15 different User eXperience (UX) properties split in 5 sets of factors explained in one of our previous paper (Abdel Razek, van Husen, Pallot, & Richir, 2018). This was inspired by previous work on UX holistic view (Pallot & Pawar, 2012) and Immersiveness factors (Pallot, Christmann, Richir, Dupont, Boly, & Morel, 2017). Each participant had to answer this bipolar questionnaire after the completion of each experiment task by using the support of each SP form. The questionnaire was devised to see how each participant emotionally, cognitively and logically perceives the applied SP form.
According to the results displayed in Fig. 10, VRSP was rated by the participants as the most responsive, which had the highest responsiveness rating from all the other SP forms. The participants also rated VRSP as very attractive and interesting, which came as a close second to ARSP for these properties. The participants also highly rated the intuitiveness, engagement, usefulness, and interactivity properties. The participants were mostly willing to recommend, and re-use VRSP, but less participants were willing to adopt VRSP in their work. This shows that the participants are willing to adopt VRSP in their work processes and more so willing to recommend others to implement and use it. As VR doesn’t offer much emotional engagement, and some may find the immersive feeling unfriendly due to a feeling of longer presence and time (Cyber-sickness). Overall, The VRSP form was ranked from good to very good in all the ratings.
As for the results displayed in Fig. 11, ARSP form appears to be the participants’ favorite among the four different SP forms. The participants rated ARSP extremely high for the intuitiveness and usefulness properties; it simply confirms the substantial advantage of having a supplementary source of information overlaying the real world objects; in order to contextually support every task completion. The participants ranked several other UX properties quite highly, including the attractiveness, pleasantness, and willingness to re-use and recommend. This shows that the participants think that ARSP has a good applicability and could have multiple uses, and that the participants are willing to re-use it and even to recommend ARSP to others. The results show also that the participants were not that attentive to the external factors; and that it didn’t emotionally engage them, and that they felt not cognitively engaged as well. The ARSP responsiveness was rated relatively low compared to the other factors; this could be attributed to the fact that the AR marker recognition was sometimes challenging for the participants; mainly because the tablet was sometimes not close enough to the marker where the participants had to move it closer to be recognized.
According to the results displayed in Fig. 12, PSP was poorly rated in almost every aspect. The participants only ranked the usefulness property as good, this can be attributed to the fact that paper has an almost infinite shelf life if handled and stored properly, and paper is very easy to use and to procure. The participants ranked the interactivity, attractiveness, and interestingness of PSP as pretty low. While almost all the other factors ranked negatively, the participants were almost neutral on the intuitiveness, willingness to re-use and willingness to adopt. This shows that participants although ranked paper quite negatively in almost all the bipolar questions, they still think it has a good value (ratio cost/benefit) in the workplace, or at least for now.
As for the results displayed in Fig. 13, the participants, especially regarding the intuitiveness and usefulness properties, rated MSP fairly high. The participants were also relatively willing to adopt, re-use, and even recommend to others, which shows that the MSP form is more appreciated than the PSP form. Although, the moderately high ranking in attractiveness, pleasantness, and responsiveness properties, the participants ranked MSP’s friendliness and attentiveness properties as neutral. The participants also ranked MSP quite low in the interactive factor, emotional engagement, and cognitive stimulation. This could be attributed to the fact that the MSP is composed of two short videos presenting the disassembling and assembling processes without any kind of audio or other form of interactivity. The video shows the optimal sequences and movements for completing the task, and it is easy to imitate as it shows another person completing the task.
Figure 14: All SP Forms Average Results (VRSP, ARSP, PSP, MSP)
Figure 14 presents the total average of each of the 4 SP forms (different colors) used in the experiment, matching against each other along the 15 rated properties. The results show that: (1) ARSP was the most highly rated overall and the highest rated in 11 of the 15 properties; (2) VRSP ranked highest in 4 properties and pretty good in the rest; (3) PSP was ranked lowest in every property; (4) MSP was ranked relatively good in 10 properties but was ranked negatively in the remaining properties. There is a clear difference between the ISP and CSP forms in the participant’s ratings and comments; in fact, it appears that ISP ranked much higher than CSP in all the properties and in the average between them. In the participants’ comments, it was apparent that the majority has enjoyed using ISP and was neutral to using CSP.
.
Figure 15 shows the UX average user ratings and it is apparent that: (1) PSP had a very low user experience average rating; (2) MSP had an almost neutral UX average rating; (3) VRSP had an almost satisfactory UX rating; (4) ARSP had a satisfactory UX rating. These UX ratings were based on the survey average ratings provided by the participants. The average score of 0.91 puts VRSP in second place after ARSP. This shows that VRSP is a viable SP form option to use in the late stages of service development; however, VRSP might require another supplementary SP form (mainly CSP) to aid in the early service development stages. ARSP was actually ranked as the best SP form by the participants if we look at the average of all the ratings which amount to a 1.01 overall. The MSP form ranked as third with an overall average rating of 0.17; the MSP form was surprisingly pretty well accepted by the participants, and has a better overall rating than expected. The PSP form ranked as dead last with an average question rating of -0.44; this was expected as it is the oldest and most outdated SP form compared to the other 3 SP forms that are based on modern technologies
Observation Results
Participants’ attitudes were observed during the completion of the disassembling and assembling tasks when each participant reacts in the stressful situation of completing these tasks. The observations made during the two experiment sessions were noted and saved in an excel sheet as following: (1) clocking with a chronometer the duration of disassembling and assembling tasks; (2) counting the number of errors; (3) counting the number of explanation requests; (4) capturing participants’ attitude (facial expressions) during the completion of the two tasks (including 3 attitude options: (a) Happiness meaning that the observer could detect a smile or positive facial expressions; (b) Neutral meaning that the observer could not decode the participant’s facial expression; (c) Frustrated meaning that the observer could see that the participant is sad or has a negative facial expression).
Figure 16: Participants Errors and Explanation Requests
The participants’ errors, and explanation requests during the tasks completion are presented in Fig. 16. It appears that: (1) only 2 participants made errors while completing the disassembling task when using VRSP showing that it was very effective in presenting the information; (2) while using ARSP and MSP, 4 participants made errors, which might attribute to some challenges in using both of the forms; (3) and only 3 participants made errors while using PSP, showing that even with the task explained in a simple traditional manner there will be mistakes as well; (4) There were 3 participants that asked explanations while using PSP; (5) Only 1 participant requested further explanation while using VRSP and ARSP; while (6) no participant requested any further explanation after watching the video while using MSP showing that imitating someone making the same task is very effective as well.
This highlights that: (1) VRSP has given the most optimal amount of information to be able to complete the two tasks with fewer errors and relatively less explanation requests; (2) MSP has implied two more participants’ errors compared to VRSP while participants have not requested any explanation after watching the video of another person completing the tasks; (3) MSP and VRSP forms could be very effective in prototyping a similar training scenarios. 70% of the participants vocally expressed their feedback concerning VRSP and MSP; where they commented on the long lasting memorization of VRSP and the short lasting memorization of MSP. This shows that a combination of MSP and VRSP might be very beneficial for covering long and short-term training processes.
Figure 17: Participants Attitudes during the Experiment
As presented in Fig. 17, there were: (1) 70% of the participants that had a positive reaction while using VRSP; (2) 43% had same positive reaction for ARSP; (3) only 20% reacted positively to PSP; (4) and 19% of the participants had positive attitude to MSP; (5) while 70% of the participants were impartial (neutral) while using the PSP and MSP, and (6) 50% of the participants where impartial whilst using ARSP, and (7) only 27% of the participants had neutral attitude towards using VRSP. The positive reaction to VRSP might be attributed to the fact that the participants enjoyed the VR environment and the interaction with the instructions simulation. ARSP had a mixed attitude, and that might be attributed to the skills needed to appropriately use the AR App, which some might have lacked.
Table 3: SP Cycles
SP Cycles |
SP Forms |
|||
|---|---|---|---|---|
Cycle 1 |
VRSP |
ARSP |
PSP |
MSP |
Cycle 2 |
ARSP |
PSP |
MSP |
VRSP |
Cycle 3 |
PSP |
MSP |
VRSP |
ARSP |
Cycle 4 |
MSP |
VRSP |
ARSP |
PSP |
The average durations in Fig. 18 represents the time that each participant needed to complete the two tasks and not the time necessary to acquire the tasks’ knowledge. The participants were aware about the clocking of the tasks, which implied that they had to try to be as effective and efficient as possible in completing the two tasks. Consequently, some of the participants were pretty engaged into the race of completing the task; while other participants had a very relaxed attitude towards the task and were not so concerned with competing or completing the task as fast as possible.
Figure 18: Average Duration for Disassembling and Assembling Tasks Per SP Form
Figure 19 shows the boxplot analysis for the disassembling tasks, we only choose the disassembly as all participants have to start with disassembly first and the average durations of the assembly were quite similar. We can see that VRSP durations are the most spread ones and the median times as well, while MSP had the smallest deviation in the sample, and smallest median as well.
The overall average task completion time shows: (1) VRSP and MSP had almost the same identical duration with a 1mn27s and 1mn26s respectively; (2) while ARSP and PSP average durations were also almost equal as well with 1mn34s and 1mn33s correspondingly. This slow task completion duration for ARSP could be attributed to the fact that the participants had to use ARSP while completing the task (AR markers show the next step after completing the initial step), while the participants used the other SP forms before starting the task. The second cycle, started by ARSP, had the average duration of 1mn34s. The third cycle with PSP as the first SP form experimented came very close to the ARSP with a 1mn33s. The fourth cycle with MSP as the first SP form used was actually the quickest in the overall average task duration with a 1mn26s. The average duration across all SP cycles and forms is 1mn30s, which is slower than the VRSP (1mn27s) and MSP (1mn26s) average durations but faster than average durations of ARSP (1mn34s) and PSP (1mn33s). While we can see that each participant’s average duration was in somehow close to each other, there is a significant difference in the participants rating between ARSP and VRSP compared to MSP and PSP. These duration differences might be due to the fact that they start with VRSP and ARSP in cycle 1 and 2 respectively, where participant might have used VR or AR for the first time and they might have faced some difficulties.
Figure 20: Disassembly Task Average Duration Per SP Cycle
Figure 20 presents the average duration of the disassembling task depending on the SP cycle used as shown in tab. 2. The first cycle, in which participants started by VRSP, had the slowest average duration of 2mn0s. The second cycle, where the participant starts with ARSP has an average duration of 01m:35s; which is significantly faster than the first cycle. The third cycle, which starts with PSP had an average duration of 01m:29s, which was also faster than cycle 1 and 2. The fourth and final cycle, which starts by using MSP had the fastest average duration with 01m:26s. The results also demonstrate that the adaptation of immersive-guided tools might be challenging depending on the user’s previous experiences, knowledge and skills.
Figure 21: Disassembly Task Average Duration Per SP Cycle Boxplot
Figure 21 shows the boxplot of the disassembly task average durations per SP cycle. The cycle 1 seems to have the largest deviation in the sample as we can see the durations vary from almost 1 minute to more than 4 minutes. MSP has the smallest, where almost all the durations vary from just below 1 minute to just above 2 minutes.
Summary of Results
Table 4 shows a summary of the observations and bipolar UX survey results, including: (a) the average number of errors made by the participants during the two tasks; (b) the average number of explanation-requests asked by the participants during the two tasks; (c) Average disassembling and re-assembling tasks duration; (d) average participants’ attitude; (e) average User eXperience (UX) rating.
These results show that there is: (1) An overall positive UX rating for the ISP (VRSP and ARSP) forms; (2) ISP forms and CSP forms (PSP and MSP) were pretty close to each other in the average duration; (3) CSP forms have scored a little lower in terms of average number of errors; (4) CSP forms have also the lowest average rating in terms of the UX rating.
One can deduct, based on tab. 1 and 4, that CSP might be more appropriate for noncomplex service processes, in which interactions are not critical, and it might be also better suitable for rapid prototyping. In contrast, ISP offers a much higher UX most probably due to a higher degree of fidelity, resolution, and interaction capacity. However, one has to bear in mind that ISP demands a much higher effort compared to CSP (e.g. paper or video-based prototypes) that do not require particular technology or training investments. Therefore, ISP appears more appropriate for prototyping complex multi-dimensional service processes (multidimensional as in containing artifacts, actors, environment, and processes) that require higher degree of interactivity.
Table 4: Summary of Observation & UX
SP Form |
(a) Errors |
(b) Requests |
(c) Duration |
(d) Attitude |
(e) UX rating |
|---|---|---|---|---|---|
(*) No SP |
4 |
4 |
11m:09s |
Frustrated |
N/A |
PSP |
3 |
3 |
03m:06s |
Neutral |
- 0.44 |
MSP |
4 |
0 |
02m:52s |
Neutral |
+0.18 |
VRSP |
2 |
1 |
02m:55s |
Positive |
+0.91 |
ARSP |
4 |
1 |
03m:09s |
Neutral |
+1.01 |
(*) No SP: is a baseline experiment, which involved 30 participants, was carried out in order to capture the duration to complete the disassembling and re-assembling tasks without any support, as well as number of errors and requests for explanation.
One can also state the following aspects: (1) ARSP has the most potential in instructional leaflet and similar informative service processes; (2) VRSP and MSP could be used in combination to cover the long and short term knowledge absorption improving training processes; (3) PSP might be used for very simple service prototyping activities but is not suitable for prototyping more complex service processes; (4) MSP is effective in delivering simple instructions for a relatively easy challenge (e.g. disassembling/re-assembling a simple mechanical element); (5) Although ISP forms ranked higher than CSP forms in several aspects, CSP forms average times were a bit faster than the ISP forms; (6) A combination of one CSP form and one ISP form could be the most appropriate solution to prototype a brand new and disruptive service; for example, using a CSP form for the early stage (e.g. co-creation) of the service development process, and an ISP for the design and implementation stages.
Limitations, Conclusion, & Future Work
This paper presents the results corresponding to a sample of 48 participants out of the overall sample of 105 participants when adding the 55 participants of the experiment sessions conducted at the University of Furtwangen. Obviously, the results presented in this paper could be seen as subject to interpretation due to the size of the sample under scrutiny. However, the preliminary results of the other sample of 55 participants exhibit quite similar outcomes. Another limitation of this study is the necessary minimum participants’ required skills to appropriately use VR and AR devices. Effectively, some participants faced challenges while using ARSP despite the briefing in introduction of each individual experiment session. For example: when a participant tries to identify and recognize the AR markers with the tablet camera. This has necessarily impacted the duration of disassembling/re-assembling. Furthermore, the success of ISP forms relies also on the participants’ level of technology acceptance; for some people there is a certain discomfort, also known as cyber-sickness, to wear a VR or AR device or adapt to its usage that necessarily impact their performance in completing a task (Pallot and Richir, 2016).
In terms of conclusion, it is obvious that any of the SP forms brings an important efficiency value when comparing the average duration where participants have no SP to support the disassembling and reassembling tasks (baseline experiment). UX survey results reveal that participants preferred ISP forms rather than CSP forms. Our empirical study also confirms that there are still some difficulties to use immersive technologies (VR, AR). The discussion about combining multiple SP forms depending on the service complexity and intricacy clearly emerged from our study results. As already explained above, CSP forms appear to be more appropriate for the early stage of service co-creation when describing a new service idea. ISP forms are revealed as more appropriate for the late stages of service development and for complex service ideas that require multiple ISP forms or even a combination of both CSP and ISP forms. CSP forms could be well utilized for non-complex service scenarios in order to save costs and time. In contrast, ISP forms require a significant financial investment and effort for implementing VR, AR or MR solutions. ISP forms are pretty suitable for multi-level and multi-dimensional service processes, especially used in the industrial manufacturing services. The use of immersive technologies in the SP process could benefit all service stakeholders by improving the user visual experience and interactivity. Obviously, it is possible to use a CSP form in combination with an ISP form; one CSP prototype form in the early stage for roughly presenting a new service, while ISP could be best used to explore the whole service scenario, analyze and optimize a service process, and related user interactions. In brief, the capacity to try a new service scenario before it gets implemented. The most important aspect of choosing an appropriate SP form is that selection has to be attuned to the SP purpose and activity set by the service stakeholders.
In terms of future work, Mixed Reality Service Prototype (MRSP) will be explored in up-coming experiment sessions in order to investigate how much an MR device (e.g. Hololens) would simplify the freely access (hands free) to service instructions compared to ARSP with a tablet. Furthermore, several industrial experiment sessions, engaging industrial stakeholders, are planned along 2019 in order to investigate how much it could be more appropriate whatever the level of complexity of the service to be co-created. However, it could be adequate in the case of investing in one MR device for training several persons; otherwise, the current device costs make it unrealistic to equip all service stakeholders. The statistical analysis (i.e. ANOVA) for validating the model and instrument will be conducted as soon as reaching some more participants.
Acknowledgments
This paper has been carried out partly in the context of the research project “dimension” that is funded BMBF (02K14A160). Authors wish to acknowledge their gratitude and appreciation to BMBF, Arts et Métiers, Furtwangen University, and all participants that contributed to this empirical study.
Notes
References
- Abdel Razek A.R., Van Husen C., Pallot M., Richir S., 2017. Innovation by Service Prototyping. In the 23rd ICE/IEEE proceedings.
- Abdel Razek A.R., Pallot M, van Husen C, Richir S., 2018. A Comparative Study on Conventional versus Immersive Service Prototyping (VR, AR, MR) VRIC '18, April 4-6, 2018, Laval, France.
- Abdel Razek A.R., van Husen C., Pallot M., Richir S., 2018. A Proposed Research Framework and Model for Service Prototyping. In 2018 IEEE International Conference on Engineering, Technology and Innovation (ICE/ITMC), pp. 1-8. IEEE.
- Arvola M., Blomkvist J., Holmlid S., Pezone G., 2012. A service walkthrough in Astrid Lindgren's footsteps. In ServDes. 2012: Service Design and Innovation Conference, Espoo, Finland, February 8-10, 2012 (pp. 21-29). Linköping University Electronic Press.
- Balzert H., 1989. CASE Systeme und Werkzeuge. Reihe “Angewandte Informatik”, Vol. 7, BI .Wissenschaftsverlag Zürich.
- Blomkvist J. and Holmlid S., 2011. Service designers on including stakeholders in service prototyping. In INCLUDE Conference proceedings.
- Blomkvist J. and Holmlid S., 2012. Service prototyping according to service design practitioners. In Conference Proceedings ServDes. 2010, Exchanging Knowledge, Linköping, Sweden, 1-3 December 2010 (No. 060, pp. 1-11). Linköping University Electronic Press.
- Boletsis C., Karahasanovic A., Fjuk A., 2017. Virtual Bodystorming: Utilizing Virtual Reality for Prototyping in Service Design. In International Conference on Augmented Reality, Virtual Reality and Computer Graphics (pp. 279-288). Springer, Cham.
- Buchenau M. and Suri J.F., 2000. Experience prototyping. In Proceedings of the 3rd conference on Designing interactive systems: processes, practices, methods, and techniques (pp. 424-433). ACM.
- Creswell J.W., Plano Clark V.L., Gutmann, M.L., Hanson W.E., 2003. Advanced mixed methods research designs. Handbook of mixed methods in social and behavioral research, 209, 240.
- Exner K., Lindow K., Buchholz, C., Stark R., 2014. Validation of product-service systems–a prototyping approach. Procedia CIRP, 16, 68-73.
- Fukuhara T., Tenmoku R., Okuma T., Ueoka R., Takehara M., Kurata T., 2014. Improving service processes based on visualization of human-behavior and POS data: A case study in a Japanese restaurant. In Serviceology for Services (pp. 3-13). Springer, Tokyo.
- ISO/IEC 20000-1 Information Technology, 2005. Service Management – Part 1: Specification, International Standards Organization, Geneva, Switzerland, 2005.
- Johnson R.B. and Onwuegbuzie A.J., 2004. Mixed methods research: A research paradigm whose time has come. Educational researcher, 33(7), 14-26.
- Jung Bae D., and Seong Leem C., 2014. A visual interactive method for service prototyping. Managing Service Quality, 24(4), 339-362.
- Krawczyk P., Topolewski M., Pallot M., 2017. Towards a reliable and valid mixed methods instrument in user eXperience studies. In Engineering, Technology and Innovation (ICE/ITMC), 2017 International Conference on (pp. 1455-1464). IEEE.
- Kuure E., Miettinen S., Alhonsuo M., 2014. Change through Service Design–Service Prototyping as a Tool for Learning and Transformation. Proceedings of Design Research Society 2014: Design's Big Debates.
- Kwon R.H., Kim K.J., Kim K.H., Hong Y.S., Kim B., 2015. Evaluating servicescape designs using a VR-based laboratory experiment: A case of a Duty-free Shop. Journal of Retailing and Consumer Services, 26, 32-40.
- Ostrom A.L., Bitner M.J., Brown S.W., Burkhard K.A., Goul M., Smith-Daniels V., ..., Rabinovich E., 2010. Moving forward and making a difference: research priorities for the science of service. Journal of service research, 13(1), 4-36.
- Ostrom A.L., Parasuraman A., Bowen D.E., Patricio L., Voss C.A., 2015. Service research priorities in a rapidly changing context. Journal of Service Research, 18(2), 127-159.
- Pallot M. and Pawar, K., 2012. A holistic model of user experience for living lab experiential design. In 2012 18th International ICE Conference on Engineering, Technology and Innovation (pp. 1-15). IEEE.
- Pallot M. and Richir S., 2016. Laval Virtual Vision 2025: Blurring the lines between Digital and Physical Worlds. Proceedings of ICDVRAT 2016, Los Angeles, USA.
- Pallot M., Christmann O., Richir S., Dupont L., Boly V., Morel L., 2017. ICE Breaking: Disentangling Factors Affecting the Performance of Immersive Co-creation Environments. In Proceedings of the Virtual Reality International Conference-Laval Virtual 2017 (p. 23). ACM.
- Peng K.L., Lin Y.L., A Nd Tseng Y.C., 2017. Constructing a 3D Multiple Mobile Medical Imaging System through Service Science, Management, Engineering and Design Systems, 5.1, 5.
- Sämann M., Abdel Razek A.R., Imran S., van Husen C., Droll C., 2016. Innovation in Prototyping for Technical Product Service Systems. Proceedings of 21st ICE IEEE.
Attachments
No supporting information for this articleArticle statistics
Views: 739
Downloads
PDF: 303
XML: 9