Exploring Mixed-methods Instruments for Performance Evaluation of Immersive Collaborative Environments

Presently, there is a clear trend for both businesses and public institutions to move towards open or collaborative innovation. Nevertheless, engaging all stakeholders, especially users, for cocreating innovative solutions and usage scenarios is, as revealed in previous studies, not so obvious. We do believe that Immersive and Collaborative Environments (ICEs) based on the use of Virtual, Augmented and Mixed Reality technologies would be the right place for co-creating, exploring, experimenting and evaluating innovative ideas and concepts in order to quickly reach a common understanding. However, there is a need to design a proper method and instruments that would allow evaluating and comparing ICEs. Our previous paper presented the outcomes of an investigation to identifying and disentangling factors characterising a group immersion and collaboration in the context of co-creation. As a step further, this paper reports about our exploratory study towards the design of mixed methods quantitative and qualitative instruments for the evaluation of Immersive and Collaborative Environments (ICE) based on previously identified factors.


INTRODUCTION
Innovation processes are more and more collective [1], [2]. Cocreation is one of the most important activities for designing innovative ideas and usage scenarios, through an iterative eXperience Design process, with users. It allows to anticipate the resulting User eXperience; hence, evaluating the potential technology acceptance and solution adoption by users. This is an essential difference with users observation-based design approaches like contextual design or even design thinking.
In this paper, we present the continuation of our previous work [3], [4] on Immersive Virtual Environments (IVE) towards the design of a specific framework for experimenting and evaluating Immersive Collaboration/Co-creation Environments (ICE). Such kind of immersive environments intended for several users has been, up-to-now, often implemented through the use of a CAVE. However, CAVE solutions are not so affordable for SMEs or Start-ups. Recently, low-cost immersive technologies have emerged, which further explains why we want to experiment and evaluate different ICE platforms that could be more affordable for small organizations. Besides discussing the research publication stream on ICE, we explore the previous work related to immersive collaboration within the context of Virtual Reality (VR), Augmented Reality (AR) and Mixed Reality (MR) technologies. Our goal is to identify and disentangle factors impacting immersive co-creation in the view of preparing appropriate experiments and related evaluation approaches.

PREVIOUS WORK
We have previously [4], [5] introduced most of the concepts used in this work, such as: Open/ Collaborative Innovation, User-Driven Innovation (UDI), Living Lab, Immersive Technologies (VR, AR, MR), eXperience Design (XD), Co-creation, Collective User eXperience (CUX), Immersive eXperience (IX), Quality of Service (QoS) and Quality of Experience (QoE). Our UDI vision fits with the Von Hippel studied "Consumer Innovation" paradigm [5]. design approaches promote distributed collaboration among organisations and user communities. However, project stakeholders are mainly trained for improving their individual skills through learning experience (i.e. practical exercises, role playing game) rather than getting a live user experience through immersive environments (e.g. Virtual Reality, Serious Games) that could unleash their creativity potential. Authors discuss issues related to the use of innovation games for enabling user cocreation in the context of collaborative innovation and experiential living labs. They also argue that people are trained for enhancing their individual skills and improving their productivity rather than for enhancing their collaboration skills and improving their interpersonal productivity. On the one hand, users or citizens have to experience something new while sharing feedback, meaning and understanding within the community; on the other hand, researchers collect data for better understanding emerging behaviours and usage patterns as well as embedded adoption mechanisms. Authors try to evaluate the role of distributed cognition and collective intelligence, including both social and emotional intelligence in the decision making process. Recently, Rihova et al. [10] introduce a Customer-to-Customer (C2C) cocreation perspective, conceptualizing the social layers within which value is formed. This approach extends current value cocreation discussions by providing conceptual insights into cocreation within customers' social sphere; hence, allows benefiting from understanding how customers can be "nudged" into more socially immersive co-creation layers.

Immersion
According to the Oxford dictionary, the term 'immersion' means "the action of immersing someone or something in a liquid" or "deep mental and social involvement in something" like the very famous "learning by immersion: a method of teaching a foreign language by the exclusive use of that language". Therefore, one could conclude that there is a notion of physical immersion, a notion of mental immersion and a notion of cultural/social immersion. According to Freina and Ott, immersive environments in education have been mainly deployed in the area of educational games due to expensiveness of immersive technology platforms and limited usability [11]. They investigate the scientific literature, over 2013-14, on the advantages and potentials of using immersive environments in Education. Brown and Cairns [12] consider that immersion is a graded experience that progresses through degrees of engagement.
As mentioned by Nordin et al. [13], [14], concepts such as fun, flow, presence and immersion have been widely used to describe the digital games user's experience. According to these authors, both Brockmyer et al. "Gaming Engagement Questionnaire" (GEQ) [15] and Jennett et al. "Immersive Experience Questionnaire" (IEQ) [16] are relevant examples of questionnaires for evaluating digital games user's experience. The IEQ has been applied in different scenarios and game types [17]- [19]. While Nordin and colleagues considered including Usability [20] in their future work, the Social Presence Questionnaire [21] that appears in their review of existing relevant questionnaires is not included.

Immersive Platform
According to Pallot et al. [22], an immersive platform is composed of both assembled immersive technologies (hardware and software) and a specific 3D content application. This kind of platform represents the technological facet of an IVE. They consider that an IVE encompasses 3 different but complementary perspectives, namely: the physical implementation of the immersive technologies, the resulting Immersive eXperience (IX) representing a specific form of UX and an evaluation framework for assessing all factors constituting the overall performance.
The IVE structure is composed of Immersive technologies (VR, AR, MR) and Immersive (3D) Content delivering an IX to users. The resulting perceived sensations are then evaluated through IX's factors. Measured QoS technical factors (e.g. Fieldof-view, frame-rate, resolution and latency factors) and evaluated QoE factors (e.g. ease-of-use, usefulness, presence) allow to assess the overall quality of the resulting IX. However, the degree of immersiveness varies also according to the type of used technologies (AR, VR, MR) and goals of the immersive application [22].
Based on reviewed publications in [22], Pallot and Richir argue that immersion represents the state of being deeply engaged (e.g. swimming), recognised as a tactical/sensory-motoric immersion; or fully absorbed in solving a problem (e.g. playing chess), seen as a strategic/cognitive immersion; or reading a captivating story or watching an exciting movie, considered as a narrative/emotional immersion. These types of immersion make one's brain so busy that everything else around simply disappears; Nordin et al. [13] refer this phenomenon to real-world dissociation. A pretty good way to measure the deepness to which a person is engaged is to observe whether time passes without being noticed and whether external world disappears. To make it short, one could argue that a 360° VR immersion bubble operates like a mind-blowing teleportation, instantaneously transporting a user in an existing remote place or a different world that is persistent enough to become another reality, even if it is a virtual one.
Innovation platforms also generate knowledge, practices and practical know-hows. The Lorraine Fab Living Lab ® , the ERPI research platform for prospective assessment of innovative usages and innovation acceptability develops "low cost" Immersive Collaborative Environments supporting front end innovation development since 2014 [4], [23]. According to our experience and our previous Living Lab (LL) projects involving various stakeholders or communities in collaborative approach [3], [24] we learnt that it is necessary to propose different degree of technological devices. Indeed, LL participants don't have the same backgrounds, knowledge and motivations. If we want to include the largest number of stakeholders in collaborative approach supported by technologies, we have to consider their different capacities to understand and use relevant new technologies. For example, Figure 1 shows the implementation for running experiments of four different immersive environments for co-creative workshops (ICEs): the Computer Screen platform (a), the collaborative interfaces on Multitouch Tables platform (b), an original collective user experiences inside one Immersive Bubble platform (c). Researchers also develop experiments with two different Head Mounted Display (HMD) platforms and its ad hoc virtual contents (d). A previous scientific paper [4] presents the technical descriptions for (b) & (c) and how all these technologies work in a complementary manner in the LF2L platform. Pictures and video are available online at: https://flic.kr/p/U114Zu. As we adopt different technological levels, we assume that these platforms don't offer the same degree of immersiveness and VR/AR/MR as well as 3D content developments. This diversity of tools reduces dependence on a specific technology and generates a higher degree of freedom to approach collectively fuzzy front end of innovation and co-creation processes.

Social Immersion
While 'Social Interaction' and 'Social Presence' are quite well established constructs in both IVE and ICE empirical studies [25]- [27], the 'Social Immersion' or 'Social Immersive' perspective is rarely mentioned excepted in the areas of Interactive Storytelling and Massive Multiplayer Online Roleplaying Games (MMORPGs) [28]- [31]. Such a social immersion or social immersive perspective characterises a 'Socially Immersive eXperience' (SIX) [31]. In the collaboration research area, the constructs of 'Interpersonal Distance' [9], [22] and 'Interpersonal Trust' [32] represent the people capacity to interact together in a virtual environment whatever barrier separates them (e.g. remote places, lack of confidence). Previous empirical studies revealed that the interpersonal or trust distance hinders the social presence. Other Scholars have been studying social inhibitors in IVE [33].

Immersion and collaboration
There are mainly two publication streams dedicated to immersive co-creation, namely: Immersive Collaborative environments (ICE) and Collaborative Virtual Environments (CVE). There are numbers of empirical studies on different immersive technology platforms that were conducted through experiments involving several participants (see Table 1).

Resulting ICE Model and Structure 3.1 ICE Model based on four Immersive Facets
Our main goal is to experiment and assess the performance of different immersive environments in the context of co-creation with users/customers/citizens. Therefore, we had to turn the IVEs into ICEs and include the multi-user context in the usage scenario. Figure 2 presents an ICE tentative model as an extension of the IVE model that includes the social immersive perspective and resulting socio-empathical factors characterising the social dimension of the resulting user experience.
There are different forms of social interaction but vocal interaction is the one that requires the highest level of participants' attention. While vocal interactions support the feeling of social presence, it makes easier the ability to reach a shared mutual understanding for each participant [41], [50]. According to Eynard, a participant of an experiment declared during the focus group debriefing [41]: "I was staying as close as possible to my team partner in the virtual environment as if I was willing to remain audible while I perfectly new that we were in fact talking to each other through the use of a headset". This empirical study demonstrated that vocal interaction tends to decrease the feeling of being present, while at the same time it increases the feeling of co-presence. For sure, all participants do not necessarily need to be perceptually immersed (sensory-motoric immersion) because of the possibility given to some of the participants to interact vocally with the one that is fully immersed and can interact with the virtual environment. It means that while participants are socially immersed and at least one of them is perceptually immersed then they can interact vocally with the immersed one for inducing actions on affordances. For example, one participant wears a HMD and all the other participants watch him evolving in the virtual environment on a large display while interacting vocally with him. The social immersion allows a group of stakeholders that are not necessarily all fully immersed to efficiently co-create together with a lower effort and cost. This is especially true for people that do not accept immersive technologies, such as people suffering from cyber-sickness (e.g. spatial disorientation).

ICE Tree Structure
Presently, the elaborated ICE structure (Figure 3) is a simple tree highlighting the two sides of the same coin, namely: the cocreation side or Collaborative eXperience (CX) and immersion side or Immersive eXperience (IX). Both include cognitive and social aspects while the IX side owns the perceptual and emotional aspects.. They are both implemented in the immersive co-creation platform and in the resulting immersive collaborative experience that appears in the Figure 2 separately in two columns.
All elements appearing in the Figure 3 are inherited from the Table 1 and are intended to lead to the selection of factors for assessing the performance of different ICEs. We don't know yet whether the co-creation and immersion facets' elements or factors are correlated or not. This is to be revealed by the up-coming experiments. It is also expected a refinement and validation of the ICE structure and model based on the results of experiments.  Table 1 and extended from [34].
The extension of the Figure 3 is characterised by the decomposition of the RWD element of the IX in three different properties. According to Jennett [38], RWD measures losing track of time, lack of awareness of surroundings (e.g. distracters) and mental transportation (e.g. a person feels completely absorbed in an activity, hence, does not feel any longer aware of being in the real world). In this ICE structure, the three properties are formalised as "timeless" for the lack of time track, "attention-less" for the lack of awareness of external surroundings events and "respond-less" for the lack of being responsive to external solicitations. Apparently, these three RWD properties depend on the extend to which one or more persons are absorbed by a common task.

METHODOLOGY 4.1 Design an exploratory experiment
Our goal is to explore what could be the most appropriate ICE model. For that, we created a tentative ICE model (see Figure 2), based on a previously described IVE model and its structure with a root, experience type (or facet), element and property levels (see Figure 3). We design an exploratory experiment based on the use of mixed methods [51], involving active participants that used our low-cost immersive technologies and then responded to a specific bipolar survey questionnaire (18 questions in Appendix 9.1).
In this experiment, the dependent variable is the collaborative experience that is measured by 8 properties while the real-world dissociation of the IX is evaluated by 1 (timeless) out of 3 properties. The independent variable is the immersive technology with three technology platforms: Computers Screen (2 workstations) platform, one Immersive Bubble platform, and one Multitouch Table platform (Figure 1). The platform d (Figure 1) will be included in the next experiment.
Our hypothesis is that these immersive technologies generate different levels of individual and collective immersive experiences. Actually, in this study we did not evaluate if participants share the same level of immersion. We rather focus on the collaborative performance of these three different immersive environments. Thus, this first experiment mainly focuses on the "Collaborative" experience. For each aspect of the four "collaboration" elements we created two bipolar questions by property, except for "Group Dynamics" property for which we made four complementary questions. Interviewees can select only one answer chosen between two antonyms (e.g. easily vs. hardly) for most questions.

Smart city context involving complementary stakeholders
Our experiment is based on a specific case study about road accidents occurring in the city. This case study involved several tasks around smart city topic. In particular, we created an experiment around a new tool to make more visual open data on road accidents entitled "Nancy'curity", as a concatenation of the city name and security. Four engineer students in computer science designed this original web-app during a "big data smart city" workshop conducted from January to March 2017. Students used HTML, JavaScript, CSS, Qjis software and GoogleMaps web interface. Using open data from French Government, this open source based app allows locating on a city map all the road accidents that occurred the previous five years in the area of Nancy Metropolis. Furthermore, this app can show the restricted speed areas: dark-blue line on Figure 4 shows speed limit (30 km/h); green, light-blue, and red lines show respectively the distances of 50 m, 100 m, and 200 m around the restricted speed areas (Figure 4).

Experimental collaborative situations
This case study consists to ask volunteers to take on the role of municipality technicians working with other stakeholders. As local technicians they have to make a road safety diagnosis from two case studies designed by the research team. The two case studies give the same kind of information: the place, the name of the road, the date and the time of one accident, and a screenshot of the local area where took place the studied accident ( Figure 5). The two screenshots were taken from the "Nancy'curity" app; finally, the only difference between the two case studies is their place in the same city.

Participants
Twelve students in psychology constituted our panel of participants: 7 female and 5 male who already know each other. This experiment was a part of their training on exploratory approach and research methods. Furthermore, we selected this group of students due to their potential interest for these Information Technology (IT) tools. Indeed, as future professionals of spatial cognition they should be able to use such tools. Immersive technologies and the LF2L research platform were new for all the participants. We divided our panel of 12 participants in four-teams of 3 participants. All teams used our immersive technologies in parallel and were involved in the similar collaborative situations in order to complete the full cycle (Immersive Bubble, Multitouch Table, Computer Screen) as described above.
After a first iteration, groups filled in the exploratory questionnaire and moved towards another immersive technology and worked on the other case study ( Table 2). At the end of the 2 nd immersion they filled in the exploratory questionnaire.  Table  2 Computer Screen 1

RESULTS
This experiment took place for about two hours and included the presentation of the research study and the indications for the experiment; the two iterations followed by individual survey; and a group discussion. Furthermore, three researchers observed and advised the participants. In the next sub-sections, we synthesize our collected data.

Findings from the questionnaire
The 12 participants filled in one form after each case exercise of the whole study. Thus we collected 24 questionnaires. We analysed the results below.

Sense Making (SMa) and Trust Building (TB)
During the exercises people shared trust, mutual confidence and understanding. Indeed, all the participants answered "easily" at SMaR.1, SMaR.2 and SMaCA.1, SMaCA.2 except one respondent who felt his/her partners were not able to explain the case using the Multitouch Table. The participants also said that they trusted their teammates (TBC.1) and question TBC.2 demonstrates that their teammates trusted them. Furthermore, the participants supported each other, except 2 participants, 1 for the two iterations, who said they were not supportive for their partners with the Multitouch Thus, "Relevance" (SMaR), "Context Awareness" (SMaCA), "Interpersonal Ties" (TBIT), and "Confidence" (TBC) seem at least maintained, even properly supported by our technology.

Shared Meanings (SMe) and Mutual Understanding (MU)
As shown in Figure 7, the results for "Knowledge Sharing" (SMeKS), Knowledge Creation (SMeKC) are less consistent that Sense making and Trust building. One participant during the 1 st iteration said that outcomes from Multitouch  Then, we study "Group Dynamics" (MUGD) through four questions. Overall, outcomes show that participants controlled the situation, except when using the Immersive Bubble platform. However, with this platform, 4 out of 6 participants' feedbacks said that the situation was out of their control, in particular during the 2 nd iteration: indeed all the answers are negative (MUGD.1). In parallel, a great majority of participant, 22 out of 24 participants said that their teammates controlled the situation (MUGD.2). Moreover, during the 1 st iteration the participants estimated that they controlled the situation in an equivalent manner with their teammates. Nevertheless, during the 2 nd iteration two participants within the Immersive Bubble platform and two participants with Multitouch Table platform indicate that  their teammates took a better control (MUGD.3). Finally, in some case with the Immersive Bubble platform, some students think that the others better controlled the situation, but the questionnaire reveals that the others admitted not control. Finally, the last question dedicated to the "Group Dynamics" aspect (MUGD. 4) shows that for all the participants the communication with their teammates was efficient.
Regardless our experimental immersive technologies, participants used "Collective Intelligence" (MUCI). Indeed, all participants in all situations shared the feeling that they collectively solved the task of the case under scrutiny (MUCI.1). However, during the two iterations with the Multitouch Table platform, one participant per group thought that she/he did not collaborate with its partners (MUCI.2). We can underline that this two participants are the same who considered being non-supportive with their partners (TBIT.2) and not so contributing to the group productions (SMeKS.2).
Outcomes of our questionnaire show that participants demonstrate goodwill to each other. During this short experiment, participants, who were lost or out of control with the technology or the case under scrutiny, trusted and relied on their group. "Mutual Understanding" can thus be understood as a "spirit of solidarity" against a new challenge. In other word, if one participant does not understand the situation or the technology, then her/his colleagues are in a different or identical situation, but they act together as a team in sharing their individual knowledge and understanding in order to reach a common understanding. Further studies could be launched for evaluating more precisely the role of Immersive Collaborative Environments in the generation of this kind of solidarity spirit.

Real-World Dissociation
Finally, the e.1.q1 question about RWD reveals different feelings: for two participants the exercises had passed slowly (Immersive Bubble then Computer); after the 2 nd iteration, three more participants had the same feeling. These three participants were in the same group operating on the Immersive Bubble platform. Thus, with 5 out of 6 participants, the Immersive Bubble platform had the worst level of RWD. According to our observations, it may be attributed to the latency generated by the technology. User has to move arms in front of a motion sensor and these gestures generate commands to Google Street Map or View (GSM/V), giving the illusion of motion along a street. Furthermore, one member of the group played with the sensor while the other members were seating inside our large inflatable bubble. This asymmetry, between one active and 2 passive participants, seems resulting in boredom for passive participants.

Findings from the group discussion
At the end of the two iterations, we organized a group discussion with all the participants based on specific questions (Appendix 9.2).
All teams really appreciated the possibility to adopt GSV besides the maps with all the technologies. This function helps to identify accidents' places; generate discussions; share knowledge and make hypothesis on the incident causes.
Participants' feedbacks also underline that several technical limits became apparent during the experimentation: (1) the tools and interface needs to be carefully checked (e.g. for one group, pinpoints didn't appear on the bird view, needing to re-launch the software); (2) strengthened or used in a more appropriate way because for example participants said that it is very difficult to read the streets' names on the bubble surface, hence, they recommended to project on a smoother and flatter material.
In fact, a maximum of five persons can be accommodated inside the bubble. In all case, only one person stays standing, using its arms to orient inside GSM/V and virtually walk along the street. The other persons are just seating on either side of the controller. These different positions seem inefficient due the passivity of seating participants. Only the participant standing at the centre of the bubble can see all the large projection, while the others are not so engaged and feel frustrated. After the experiment, participants suggested that a computer mouse should replace the motion sensing input device. In other word, they were interested by the size of the projection because a large display generates a higher feeling of immersion. However, the "shared" virtual walk did not convince them.
Finally, participants were more collaborating for understanding and learning how to use appropriately the Immersive Bubble platform than solving the incident case under scrutiny. Teams felt that they were more talking about the platform than they talked about the incident case due to the inconvenience of the immersive bubble technology. It became obvious when comparing with the participants' feedback about the Multitouch Table platform that was much more engaging all of them. This platform appears as providing a better feeling of immersion due to the size of the display and that all participants were able to see the same elements of the incident case. Finally, the panel of participants questioned the real impact on teams' production in comparison to the traditional Computer Screen. Nevertheless, the participants really appreciated to draw and co-create using virtual 3D objects with the experimental multitouch interface. "This is an asset that we can draw," said one student. Indeed, according the Multitouch Table potential, we developed an original application including "Nancy'curity" and offering advanced navigation as well as interaction capabilities. Using a digital experience creation platform, we built an interactive experience allowing selecting street-views of the accident's place and customize this view drawing with fingers or adding 3D objects (cars, trucks, houses, buildings, trees, etc.) Every team has used this digital tool to illustrate or reproduce the accident and co-created new elements for improving road safety (Figure 8). Table  platform and its experimental interface.

Figure 8. One co-creation generated with Multitouch
The group discussion also developed beyond immersive technology. For example, participants suggested investigating how the possession of a driver's license (i.e. a knowledge of the driving rules and driving experience) could impact the way of solving a specific incident case. Furthermore, they asked the same question about the knowledge of the place like how a real experience of the road environment helps to understand and analyse an incident case.

Outstanding issues
We assume that the exploratory aspect of our questionnaire and our experiment through iterations between the elements generated new knowledge and specific questions that we explored in order to strengthen ICE evaluation model and metrics.
How time and the technical expertise influence the degree of collaboration? Can we observe more collaboration or more tasks distribution through the time? Can we improve our exercise and put the collaboration as the key to the success? How to design a more complex situation in which the more one collaborate, the more one succeed?
How can we enrich our metrics? Which indicators can be used to indirectly measure the ICE impact on collaboration? E.g. curiosity as parameter of efficient collaboration: Is group curiosity increasing, as indicated by a larger amount of accurate data? In this case we could compare the qualitative group productions (text and picture). In our case, these data are not relevant due to the low level of complexity of our case study.
As evaluation and recognition of weighting team results, Rigby et al. [52] suggest "shifting metrics from output and utilization rates (how busy people are) to business outcomes and team happiness (how valuable and engaged people are)." We also need to experiment our ICE model with a greater diversity of participants, reflecting the multidisciplinary context of organization. This aspect will also contribute to evaluate the ICE model and structure.
According to the group discussion and our observations, one aspect of the experimental protocol could be easily improved. It seems for this kind of experiment that it is relevant to give some advises before starting with the multitouch interface and the other immersive technologies; and to let the participants to do a dry run of the new equipment for few minutes.

CONCLUSION AND PERSPECTIVES
In a first step, we have created an ICE model, its structure and metrics based on our own previous work and through a specific literature review for identifying the most relevant elements and selecting the most appropriate ones for the context of immersive collaborative environments. In a second step, we have designed an exploratory instrument and experiment conducted on LF2L platform. Thus, this study highlights the interdependency and complementarity between quantitative and qualitative instruments leading to apply a mixed-methods approach.
As a conclusion of this empirical study, though it is still on-going work, we have identified a fourth facet to the construct of immersion that is "social immersion". However, the social aspect was already perceived in the previous IVE model through the socio-emotional and socio-cognitive immersion facets. These two facets represented the potential group behaviour and collective intelligence that could already be observed in multi-user immersive games.
Furthermore, for future studies and experiments, in particular with SMEs and professionals, we have to take into account the maturity of the ecosystem (technology, usages, business, internal organization, partnership, etc.) Indeed, the development of an experimental metrology based on mixed-methods for the ICE performance depends on several interconnected dimensions. Moreover, this exploratory research focuses on short term. We only evaluated the direct impact of ICE on immediate collaboration. We can talk about direct profit for users. This is the initial value generated by ICE. It seems other values could be identified and integrated in our metrology as the "strategic value", derived from the connections that participants establish with one another and from further collaborations; or the "Exit value" defined by learnt knowledge, components, or solutions, tapped by participants in the future. These values are adapted from [2].
Finally, as future work, we will continue to reconcile the immersive and collaborative experience. The outcomes of the upcoming experiments will allow refining and eventually validating totally or partially our proposed ICE model, structure and metrics. 10. BIOGRAPHY Laurent Dupont, Eng. PhD. is senior researcher at ERPI (Research Team on innovative Processes, Université de Lorraine, France), lecturer at ENSGSI and visiting lecturer at TELECOM Nancy. He is the cofounder (2014) and scientific manager of the Lorraine Fab Living Lab ® (www.LF2L.fr), the ERPI research platform for prospective assessment of innovative usages and innovation acceptability. He is also the co-designer and scientific coordinator of "Lorraine Smart Cities Living Lab" (ENoLL member since 2010), an interdisciplinary project involving several laboratories and other public and private partners. He designs, implements and evaluates new processes, based on collaborative innovation involving users, companies and territories, generating smart and sustainable Cities. Contact: l.dupont@univ-lorraine.fr Marc Pallot, PhD.

ACKNOWLEDGMENTS
University of Nottingham. Marc spent about 20 years in the industry with different positions and responsibilities on innovation management, product design, product marketing & engineering in different sectors: Telecom, IT, Defence, and Aeronautics. He was also a serial entrepreneur acting as co-founder of several start-ups. Since 2000, he is a senior research associate at the Centre for Concurrent Enterprise, Nottingham University Business School, and at Arts et Métiers