Articles

A Systematic Literature Review on Extended Reality: Virtual, Augmented and Mixed Reality in Working Life

Abstract

Extended reality (XR), here jointly referring to virtual, augmented, and mixed (VR, AR, MR) reality, is becoming more common in everyday working life. This paper presents a systematic literature review of academic publications on XR indicating changes in practical organization of work. We analyse both application areas of XR and theoretical and methodological approaches of XR research. The review process followed the PRISMA statement. Design, remote collaboration, and training were the main application areas of XR. XR enabled overcoming of obstacles set by time and space, safety, and resources by mediating experience of space. Research on XR applications in actual working life settings is yet relatively rare and covers primarily three areas: collaboration, evaluation of knowledge transfer, and work practices. Virtual reality was the most common form of applied XR, although the hardware used varied case by case. We identified four research areas regarding XR: collaboration, work practices, and evaluation of knowledge transfer, which somewhat followed the application areas. We did not find XR-specific methodologies in the reviewed articles, only few recent studies used novel ways of collecting research material, such as recording the movement in virtual reality. For now, XR still holds significant potential rather than clearly confirmed general advantages in working life.

1. Introduction

Extended reality (XR), here referring jointly to virtual (VR), augmented (AR) and mixed (MR) reality, has become more common in working life during the previous three decades. Companies are investing more and more in various XR applications as user-friendliness and availability of these technologies grow, and investments in these technologies are often expected to enhance both work efficiency and productivity (e.g. Du et al., 2018; Miettinen and Paavola, 2014; Stanton et al., 2020). Following the technical development of XR technologies, the amount of research has also increased (Radianti et al., 2020; Wang et al., 2018).

The increasing number of XR publications from different perspectives require an overview of the current state of the art. Previous reviews have concentrated on specific forms of XR, that is, either VR, AR or MR, or even specific devices, such as VR headsets (e.g. Radianti et al. (2020), Zahabi and Abdul Razak (2020)). While this approach allows to study a specific technology in detail, it does not provide a general overview of theoretical and methodological approaches used XR research in actual working life. Most individual studies (see e.g. (Seymour et al., 2018 ; Yan et al., 2011)) have concentrated on technical development, user experience and scenario pilots of XR applications rather than social and practical uses (see e.g. Nikolić et al. (2019)).

We looked for changes in practical organization of varying working life contexts, since fewer studies have focused on added value in concrete ‘real-life' uses of this technology (Berg and Vance (2017), p. 4; Maftei and Harty (2015), p. 53-54). Medical area was excluded from the review since it has specific features on the use of XR and requires own reviews. We are interested in XR research from social science perspective which is needed for having a multidisciplinary perspective and for understanding and developing practical uses of XR in organizations (Fast-Berglund et al., 2018; Fox et al., 2009; Pan and Hamilton, 2018; Riva, 1999).

Our aim is to search and analyse research on practically oriented uses, and potential benefits of XR in working life collaboration. We are interested in how XR potentially transforms collaborative ways of working, or ways of organizing specific work anew. New technologies are not just important from individual worker's point of view, but they often transform ways of organizing work and working together (Orlikowski, 2007). Social, educational, and organizational implications of XR applications to working life cannot be researched separately from each other. Implementation of new technologies usually requires long term evolution and development (Miettinen and Paavola, 2014). Short-term pilot experiments advance development of XR (e.g. Stanton et al. (2020)). However, they fail to address the potential of already existing XR in contemporary working life and there is a need for wider cumulation of practical knowledge (Radianti et al., 2020). What kind of ways of working existing XR advance or enable in working life? What kind of issues or challenges are solved with the aid of XR rather than created alongside its development (Madary and Metzinger, 2016)? We identify relevant characteristics of XR for collaborative ways of organizing work by analysing existing studies and try to find answers from the perspective of social sciences to assemble practically applicable, research-based knowledge. Additionally, this review is complementary to technology-focused studies and profit development of uses of XR technologies in practical settings.

We analysed academic journal articles published between 2009 - 2020 on the application of VR, AR, and MR, referred together as XR, in working life and organizational collaboration. We utilized a PRISMA statement and checklist for the sample search process and additional inclusion criteria of articles (Moher et al., 2009). The article proceeds in the following manner. Section 2 provides background for the review including the discussion of the basic definitions of XR and an analytical description of the previous reviews in the field. In the Section 3, we describe the methodological framework for the study, including the description of the data collection and data analysis procedure. The Section 4 contains the results of the review. The discussion and remarks on limitations of the article are in the Section 5 and we conclude our article in the Section 6 by outlining potential further research lines of XR from the perspective of human and social sciences.

2. Background

2.1 Ways of defining virtual, augmented, and mixed reality

In this article, we use the term extended reality (XR) as an umbrella term for diverse kinds of technologies. XR is a relatively loosely defined concept, referred also as “cross-reality” (Sherman and Craig (2019), pp. xxiv & 19). XR covers all forms of virtual, augmented, and mixed reality (Fast-Berglund et al. 2018). As the limits of different XR forms are ambiguous and our review focuses more on the practical utilization of this technology rather than the technology itself , XR is the most appropriate concept for this goal. Occasionally virtual (VR), augmented (AR) or mixed reality (MR) are used in the article for accuracy instead of XR, as most previous sources do not use the XR term, but more specific forms of it. Our aim is not to present a specific taxonomy of XR technologies. As social scientists, we are more interested in the ways of practitioners define these technologies and the review aims to provide some groundwork for further understanding on these technologies from the perspective of practical uses and collaboration. For more detailed technical account on the relation of XR technologies we recommend for instance the paper of Mann et al. (2018).

There are many ways of defining XR technologies. First conceptualizations and ideas on VR were developed already at the 1960s (Aukstakalnis, 2017; Berg and Vance, 2017; Whyte and Nikolić, 2018). Further development of VR applications started at the 1990s as multiple technological solutions advanced (see e.g. (Moore, 1995; Moore, 1998)). Milgram and Kishino's (1994) classic virtuality continuum have often been used as a basic categorization of VR. According to the virtuality continuum, there is a mixed reality between real-world environments and virtual environments. In VR, a participant-observer is totally immersed in a synthetic, digitally produced world. The concept of AR came into use in the 1990s (Aukstakalnis, 2017, p. 2). Generally, AR refers to otherwise real environment, which then is “augmented” with digitally produced virtual objects. Its counterpart augmented virtuality refers to cases where fully virtually produced environment is augmented with the elements of ‘real' environment (Milgram and Kishino, 1994, pp. 2-4). According to Azuma (1997, p.2), the crucial difference between VR and AR is that “AR supplements reality, rather than completely replacing it.” Generally, contemporary MR is established to refer to augmented reality and augmented virtuality together (Fast-Berglund et al., 2018), thus acting as synonym to augmented reality. Sherman and Craig (2019) would include AR as a part of VR. However, some of the recent studies see AR and VR as separate “sibling” technologies (Wang et al., 2018). The discussion on the correct usage and referential relationships of concepts is overlapping with the continuous advancement and new applications of technology.

Regarding this on-going discussion of XR conceptualization, in this review we are starting with immersion and interactivity as two central features of XR (see Radianti et al. (2020); Sherman and Craig (2019)), and we are looking at how they were treated in the reviewed articles. Depending on the exact form of XR and utilised devices, interaction and immersion have slightly different meanings. We treat VR and XR first and foremost as a medium, and virtual world or virtual environment as the content of this medium (Sherman and Craig, 2019). Immersion and interaction are characteristics of the VR/XR medium. Other characteristics are participants, creators, and virtual world, i.e. the content. All XR applications have glimpses of virtual worlds, which can be fully immersive environments or mere virtual, digitally produced objects superimposed into the ‘real' world.

Immersion refers to the “sensation of being in an environment” which can be accomplished through mental or physical means (Sherman and Craig, 2019, p.10). Slater and Wilbur (1997) presented a framework for immersive virtual environments and saw presence as a state of consciousness relating to immersion. Nykänen et al. (2020) examined the sense of presence as a consequence of immersion. Immersion can then be considered as one core character of XR (e.g. Radianti et al. (2020, pp. 146-147). The physical means enable the bodily sensation of entering a virtual world as the content of VR, a mere medium. In other words, physically or technically approached immersion is exclusion of the “real” world to sense the digitally produced one (Wang et al., 2018). VR is often described as a provider of an immersive experience (Whyte and Nikolić, 2018, p. 3). Immersive experience is close to sense of presence (Berg and Vance, 2017, pp. 2-3).

There can be different extents of immersion, from fully immersive systems to non-immersive ones (Jamei et al., 2017). Wang et al. (2018) as well as Zahabi and Razak (2020) define immersive VR through the hardware it is accessed: desktop-based VR is non-immersive whereas VR accessed through headsets is immersive. Whyte and Nikolić (2018, p. 25) refer to non-immersive systems as standard monitors or 3D glasses and raise up the question whether these are VR systems at all, due to the lack of immersion. With AR, immersion is considered rather as a feature of usability (Aukstakalnis, 2017). The ideal usability is achieved when the use of technology is effortless and do not cause additional breaks during work. From the point of practical organizing of work, this might be the most fruitful conceptualization of immersion since it is easily observable.

Interaction or interactivity may refer either to user's interaction with the medium (XR, virtual environment or virtual objects in real environment) or to interaction between users with the aid of the medium (Pan and Hamilton, 2018). Even with the latter option of interaction between the user and the medium, interactivity may refer to mere navigational possibilities: that the user is not constrained to predetermined viewpoints but can position themselves in the environment and change perspectives (Berg and Vance, 2017, p. 3; Whyte and Nikolić, 2018, pp. 20-21). More advanced XR applications allow the modification of VR environment in real time (Radianti et al., 2020, p. 3), and the virtual environment can consist of “physical” manipulators mimicking features such as lighting, vibration, wind, temperature, and pressure, which then can be changed (Berg and Vance, 2017, p. 2).

2.2 Previous reviews of extended reality in relation to collaborative work and learning

There are several previous reviews from the point of view of collaboration or learning regarding different forms of extended reality: Mikropoulos and Natsis' (2011) review on educational virtual environments, Jensen and Konradsen's (2018) review on VR head-mounted displays (HDMs), or simply headsets, in education and training, Radianti et al.'s (2020) worked equally on immersive VR HMD's in higher education and Wang et al.'s (2018) review on the use of VR in construction engineering education and training. In addition, there are several field specific reviews, e.g. in construction (Kokkonen and Alin, 2015; Li et al., 2018, Zahabi and Razak, 2020).

Mikropoulos and Natsis’ (2011) review concentrated on articles published during 1999 – 2009. The authors pointed that VR offers possibilities for multisensory interaction, while visual presentation is still dominant source of information for the user, and that interactivity had a minor role at that stage of technology development. In addition, they indicated that the feeling of presence is an important potential aspect of VR.

Concurring with Mikropoulos and Natsis, Jensen and Konradsen (2018) perceived immersion and presence as the key concepts of VR in their recent study. Jensen and Konradsen concluded that headsets’ use does not automatically guarantee learning, but it enables access to “situations that are either inaccessible (in time or space) or problematic (dangerous or unethical)” (2018, p. 1517). They asserted that the future research on VR headsets should concentrate on “how and for what” the technology should be used (2018, p. 1526), since it is only matter of time for it to be vastly applied.

Radianti et al. (2020) continued the work of Jensen and Konradsen (2018) by studying VR headsets from all price ranges. They concentrated on technologies as immersive as possible, thus excluding desktop-based VR interfaces, CAVEs and panoramic (360) videos. Most of the articles in this review did not describe any learning theory at all and focused on the usability of technology, while in others learning theories were disconnected from the actual use of technology. As a common design element, Radianti et al. (2020) identified interaction and divided it to interactions with the VR environment and interactions with the hardware. Similarly to Jensen and Konradsen, they also suggested broader studies on VR in education and claimed that “proposing a taxonomy of learning theories and other framing factors for educational VR applications is a future research task” (2020, p. 26).

Wang et al. (2018) had a specific field with VR application in their review, as they analysed studies on construction engineering education and training (CEET). The authors examined virtual reality through field-specific history, and represented VR as a visualization technology, which can be integrated with building information modelling (BIM) and thus replacing computer-aided design (CAD). They identified several types of VR applied in CEET. The authors also emphasized the potential of VR to transform learning from teacher-centred to student-centred, thus increasing the autonomy of an individual.

The presented reviews highlighted VR technologies, not using the term XR. Collectively, they present the ambiguity of distinction of VR, AR, and MR and underline the potential immersive experience that VR offers in collaborative contexts. They were oriented on uses at pedagogical or educational contexts, but not in working life, which is our focus. In our review, we included all forms of XR, and we are interested in the aspects of immersion and interactivity. Unlike Radianti et al. (2020, p. 21), we let the authors define the meaning of each utilized term and did not exclude articles identifying presumably “less immersive” technology as VR. Instead, we wanted to make various research approaches towards this new technology field visible.

3. Research design

3.1 Research aim and questions

Our research aim is to inquire research on application areas and use of XR in working life collaboration. In addition, as the technology is still emerging, we are interested in its key characteristics affecting the use and research. Our focus is on practically oriented research. In addition, the objective is to specify, what kind of theoretical and methodological approaches research on XR in real-life settings has been applied. We put forward the following research questions concerning actual working life collaboration:

What are the main application areas of XR? What are its potential added value and limitations?

What kind of XR applications and devices are utilized? What characteristics are emphasized in their use?

What theories or key concepts, and methodologies or methods have been applied in analysed articles?

We wanted to focus on utilization of XR in actual working life instead of pilots or experiments, but we expanded the scope somewhat as the number of articles that fulfilled the criteria appeared to be small. Still, the emphasis was put on empirical research articles targeted at the actual collaborative uses of XR instead of piloting of entirely novel solutions of technology development. We also accepted some papers close to piloting if they met the other criteria (e.g. (Hallbeck et al., 2010; Shen et al., 2013) We did not include reviews and theoretical papers in the analysis but some of them (e.g. (Radianti et al., 2020)) appeared in the search and we utilized them for framing the background.

3.2 Research method

We conducted a systematic literature review addressing knowledge gaps (Dawidowicz, 2010; Van Wee and Banister, 2016) of application, features, and research of XR in collaborative, real-life settings. The additional value of this kind of review comes from the comparison of methodologies, gaining novel knowledge and knowledge synthetization for future research and for practical utilization of XR. The generality of the research assignment could point at categorizing this also as a scoping review but more importantly we have applied the methodology and recommendations for conducting a systematic review (Arksey and O'Malley, 2003; Booth et al., 2016; Moher et al., 2009).

The lack of specific methods for conducting a literature review is generally addressed as their weakness (Booth et al., 2016; Van Wee and Banister, 2016). In this review, Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) statement (Moher et al., 2015; Moher et al., 2009) was the core method for reporting each step of a review process from the sample search to final analysis for solving this ambiguity of review methodology. Although the PRISMA statement originates from the needs of medical sciences and health care, it is an appropriate tool for systematizing review making process in other research fields as well (Moher et al., 2015).

3.3 Data collection procedure

3.3.1 Search engines

We conducted the sample search within three different scientific databases: Ebscohost (EBSCO), Elsevier's Scopus and Web of Science maintained by Clarivate Analytics. These three were chosen based on evaluations in academic articles (Adriaanse and Rensleigh, 2013), search modification and recording possibilities of each search engine.1 The revision of search engines was vital, since they limit and direct possible search results and thereby the conclusions of the whole review. For instance, Google Scholar is a good source for grey literature, such as conference papers, academic books, and other publications, adding depth to reviews (Haddaway et al., 2015; Paez, 2017). We excluded grey literature from this review, so Google Scholar was not used.

3.3.2 Search criteria

The review considered solely peer-reviewed journal articles in English. This was the first step for verifying the quality of included articles (Dawidowicz, 2010). Search criteria were modified multiple times, as multiple details affected the search results. Essentially, the three search engines were chosen to achieve as comprehensive search results as possible. The fluidity of XR as a concept resulted to tens of thousands of articles, which needed more targeting. The search criteria were modified to emphasize practically oriented research on actual use of extended reality in working life. Conclusively, three different keyword groups were formed.

The first keyword set (virtual reality OR augmented reality OR extended reality OR mixed reality) covered the technology itself including all XR variations from virtual, augmented and mixed to extended reality. Search with abbreviations, such as “MR”, were tested but it did not significantly change the search results, acquired with full words. In some cases, abbreviations added irrelevant articles into search results, as especially “AR” can be simply a part of a word. Moreover, we maintained the word “reality” with its counterpart extended, virtual, augmented, or mixed, although there are multiple variations. For instance, Mikropoulos and Natsis (2011) use the term virtual environment in their review, but contemplated, in fact, on same issues when virtual reality medium is in question. Cave automatic environments (CAVE) were not included in the keyword set as we presumed research articles within our scope would use some form of XR in addition to CAVE in their topics, keywords, abstract or title.

The second keyword set (participat* OR collaborati* OR coordinati* OR co-operati*) was formed in order to concentrate on the actual use of extended reality in working life settings requiring and/or transforming collaboration and cooperation. We were not interested in how XR transforms individual working tasks, but how it potentially transforms overall collaboration between people and other technological and material objects. Since our aim was to collect research on actual working life collaboration, we took special interest in social and educational research and looked for how technology shapes and changes how the actual work is organized and conducted (Orlikowski, 2007). Therefore, we searched for articles where social sciences were contributing to XR research in different research fields or as a part of multidisciplinary research. Therefore, these keywords (participat* OR collaborati* OR coordinati* OR co-operati*) representing quite wide scope of interest were addressed to this category to obtain as wide range of articles as possible.

The third keyword set (work* OR organization* OR organisation* OR occupation*) aimed to limit the search results specifically to working life context and to organizational transformations. This was one of the most challenging part of the sample search process. The research represented by selected articles did not need to be specified as work or organizational research as long as the context for XR application was working life. Conclusively, this keyword set included somewhat different terms (work, organization, occupation).

3.4 Data analysis

Article identification

We conducted the article identification process twice: first in January 2020 for articles published between 2009-2019 and second in August 2021 for articles published during 2020.2 Keywords were searched in article titles, abstracts or in article topics. Based on sample searches, a keyword search limited to only titles and keywords of an article would have been too restrictive. Words related to the primary school or strictly medical context were excluded. The final search string was following:

(virtual reality OR augmented reality OR extended reality OR mixed reality) AND (participat* OR collaborati* OR coordinati* OR co-operati*) AND (work* OR organization* OR organisation* OR occupation*) AND NOT (medic* OR chemi* OR school* OR child* OR therap*)

The first search between 2009-2019 brought 62 results from Ebscohost’s Academic Search Complete, 31 results from Scopus and 213 search results from Web of Science.3 Altogether there were 306 results from the three search engines, and we added one article from other sources (Maftei and Harty, 2015) for the screening, leading to a total of 307 articles.4 After duplicates were removed the result was 284 articles. The second search brought 10 results from Ebscohost’s Academic Search Complete, 80 results from Scopus and 89 search results from Web of Science, concluding as 179 results altogether. After duplicates were removed, the search resulted to 149 articles. These two searches united, data analysis is reported phase by phase in Prisma Flow Diagram (Image 1).5

Image 1. Prisma Flow Diagram.

Screening

Altogether we identified 485 records through academic search engines and one from other sources, as the work was already familiar to us. After duplicate were removed there were 433 records. There were two search phases, first in January 2020 for articles published between 2009-2019 and second in August 2021 for articles published in 2020. First 307 results were recorded into an Excel spreadsheet with the reference details. After the duplicates were removed 283 articles remained for the screening. The second search for articles published in 2020 was recorded straight to Endnote reference software and included altogether 149 articles for screening after the duplicates were removed from the total of 179 articles.

The screening was conducted by the first author of the article. During the screening, article abstracts were read through carefully considering their applicability for further inquiry with the following criteria:

Article is a peer-reviewed journal article in English published 2009-2020.

Article is based on research material that involves utilization of a VR, AR, MR or XR tool and the tool and its purpose are defined.

In the empirical part of an article, XR is either already applied or at the experimental phase of working life application.

Article’s qualitative and/or quantitative research material is at least partly gained through empirical research from either collaborative or cooperation in working life context.

The screening process and reasons for exclusion for each article were recorded into the Excel spreadsheet in the first phase, and to Endnote in the second phase. As previously mentioned, we needed to stretch our requirements on the third criteria. Instead of concentrating strictly to research considering XR in settled use, we included articles that represented XR cases aimed at aiding real-life challenges in working life even though experimentally. Otherwise, the search results would have been rather narrow for the analysis, and it seems extended reality applications, or at least the research on them, are still mainly focused on novel and experimental solutions.

The data analysis method followed critical systematic literature review. Van Wee and Banister (2016, p.284) presented in their paper “stratified selection” to apply when a review would otherwise include too many articles. This selection process was partly overlapping with analysis of the papers, since deeper understanding of contents of articles grew as the review proceeded. Contemplation and re-evaluation of the research setting were common to the analysis phase.

Eligibility

The number of full-text articles assessed for eligibility was 75 after previously shown requirements. Eight articles were excluded due to accessibility issues although full-text availability was one of the boundary rules of search criteria when possible.

Included

From 75 articles addressed for the full reading, we accepted 26 for the analysis phase. At this phase, the additional exclusion criteria were following:

No applicable empirical research material or empirically shallow (19)

Development, prototyping or testing (18)

Not related to working life (8)

Inaccessible (8)

Does not include any application of extended reality (5)

The greater number of articles in the parentheses of exclusion criteria (58) than in de facto excluded (49) assign that some of the excluded articles did not meet our criteria in several ways. The result might have differed regarding the reviewers of articles. For this reason and to minimize the bias, two authors separately read the 75 articles to ensure the clarity and validity of the described selection criteria and selected 26 articles for the final analysis meeting the criteria.

In sum, three different search engines found 433 articles, of which 26 articles qualified for the analysis. For the data analysis, we used Excel spreadsheets formed during the data collection process and Endnote. More detailed information on the articles were added to the file, such as theoretical background, methodology, methods, and specific features of utilized technology in each article. The results are organized in three sections according to the research questions.

4 Results

4.1. Application areas, potential added value, and limitations of XR

4.1.1 Application areas of XR

The analysed articles as well as the application areas are specified in Table 1. We identified three main application areas of XR in the articles: design (13), remote collaboration (6) and training (5). In addition to the three themes, category ‘other' includes two cases: Haavik (2016) inquired AR tools in surgeons' work and Barreau et al. (2015) covered a VR experience of an 18th century merchant ship built for historical analysis. We excluded medical area from our search, as uses of XR in medical field constitute a specific, own field of research with its specific questions and emphases (see e.g. Davids et al. (2020)). Haavik (2016) is an exception which was included in this review since it met the search criteria and focuses specifically on work practices of surgeons rather than medical uses of XR.

Under the category of design, urban planning was well represented covering five out of thirteen articles (Bratteteig and Wagner, 2012; Dembski et al., 2020; Sanchez-Sepulveda et al., 2019; Wagner, 2012; Wang and Chen, 2009). Two articles (Bratteteig and Wagner ,2012; Wagner ,2012) analysed the same three case studies utilizing MR set-up called MRTent for participatory design of different urban areas in different countries. In addition to five urban planning articles, six articles were in automation, engineering, and construction (AEC) sector. For instance, XR enabled collaborative design of a hospital building (Maftei and Harty, 2015) and worked as an enhanced communication tool between designers and customers overall in AEC (Shen et al., 2013). Hallbeck et al. (2010) inquired the use of XR in an efficient workplace design in five different industrial settings. Aldabra et al. (2020) used VR in design workshops for refugee camps.

XR was applied for remote collaboration in six cases. Remote collaboration cases were in remote work meetings (Da Silva and Garcia, 2013; Hayashida ,2018; Oprean et al., 2018), design-related tasks (Narasimha et al., 2019) and in security services, such as by military personnel or security guards (Lukosch et al., 2015; Nilsson et al., 2011). The studies of Da Silva and Garcia (2013), Hayashida (2018), and Oprean et al. (2018) concentrated especially on the spatial experience that can be shared and provided remotely. The studies support the idea that shared spatial experience strengthen the feeling of presence and thus may support the more general feeling of shared social experience.

The five articles considering training included both individual training scenarios (Anastassova and Burkhardt, 2009; Nykänen et al., 2020) and team training (Bertram et al., 2015; Nilsson et al., 2011; Passos et al., 2017). The sub-area of team training was partly overlapping with remote collaboration, as team members often were in remote locations. Bertram et al. (2015) noted that while knowledge transfer was measurably the same with training in physical and virtual environments, trainees were more motivated in physical environments. The research of Nykänen et al. (2020) found the perceived self-safety motivation higher in VR-based training versus lecture-based option. While this would suggest the realistic physical scenario to be most efficient and VR-based training better than mere lectures, the differing research settings and manners of measurement do not justify strict comparison.

Table 1. Analysed articles and application areas of XR.

As noted in the description of data collection an analysis, we needed to stretch our original requirements for long-term real-life uses of XR. Although technological development has been significant and possibilities of XR keep widening, the practical uses still seem rare based on our results.

4.1.2 Potential added value of extended reality in working life

We found two ways in which XR technologies are at the moment contributing into working life in practical settings compared to potential technological promises of them. First, extended reality can potentially enable scenarios otherwise inaccessible or unreachable. The reason for inaccessibility may exist due to costs or danger of effort (Bertram et al., 2015). Safety was especially emphasized in the training of security personnel (Lukosch et al., 2015; Passos et al., 2017), construction workers (Nykänen et al., 2020), and underground coalminers (Grabowski and Jankowski, 2015). As another example of increased safety, Haavik (2016) inquired unspecified AR tools use in surgical operations to access to internal body parts with minimum disruption. In addition, accessing previously inaccessible could mean accessing space-located knowledge in real time (Anastassova and Burkhardt, 2009). For instance, to Barreau et al. (2015), the technical realization of historical and naval information on virtual sailing enabled historical analysis from multiple aspects, i.e. general living conditions and social actions.

Second, the articles stressed the potential of XR to improve already existing practices. For instance, it may offer additional possibilities “as a tool for advice and support in stressful situations” (Lukosch et al., 2015, p.636). In addition, XR can offer more adept visual content for the design process (Bratteteig and Wagner, 2012; Gül, 2018; Maftei and Harty, 2015; Shen et al., 2013) by enabling rapid modification of a design model and multiple revisions. This may ease already existing practices or, by becoming more cost-efficient, enable it in some cases, where the budget might not have previously allowed it. For instance, in the article of Bertram et al. (2015) an adaptive desktop VR application called ViPol was used for team training of cooperation between firefighters and police officers in dangerous situations. Normally, the costs of using a helicopter rescue services in training situations would be too high.

Training scenarios such as practicing the removal of undetonated explosives (Grabowski and Jankowski, 2015), or evaluating socio-cultural appropriateness of designed refugee camps (Albadra et al., 2020) or harshness of living in a 18th century ship (Barreau et al., 2015) illustrate how XR technology has the potential of overcoming multiple previously unsurpassable obstacles set by time and space, safety, and resources. As a distinct feature from other digital technologies, the possibilities of XR bring forth novel spatial experiences. In an especially detailed manner, the article of Barreau et al. (2015) brings up how analysis of historical living conditions is enabled in a manner otherwise impossible, as the virtual reality modelled the movement of ship, cohabitants, and weather conditions. The literal ability to set yourself in this environment not allowing to forget the conditions in their entirety may pose one of the main strengths of XR: a wholesome shareable experience.

4.1.3 Limitations of XR in actual working life collaboration

Limitations of XR were primarily related to technical usability issues, undeveloped practices of technology applications, and lack of resources. Many of these limitations were the result of the novelty of XR. We had difficulties finding cases of long-term use, the novelty may have an accentuated position as an explanation for limitations. However, the virtual model of a historical ship (Barreau et al., 2015) was an exception, as it was utilized in multiple occasions over the years. In addition, some novel models at the time of the article writing might have been applied later on in long-term use, such as virtual safety training developed for multiple construction businesses in Finland (Nykänen et al., 2020).

Limited accuracy (Hallbeck et al., 2010; Sanchez-Sepulveda et al., 2019), eyesight of XR headsets and relatively heavy and bothersome equipment to carry (Lukosch et al., 2015) counted as technical usability issues, that technical development may ease and remove. There is no patent solution for movement in virtual worlds but case by case technical limitations can be bypassed. Alongside a keyboard and a mouse, handheld devices were the most common way of moving in a fully virtual environment, although it was not specified in all of the articles. Shen et al. (2013) underlined that integration of human movement in VR could equally increase the understanding of the design in question. Passos et al. (2017, p. 327) noted that security training in fully virtual training environment would have been more efficient, if the environment would have “the possibility to have a variable number of agents” in it imitating unpredictable human actions, i.e. virtual world would include more interactive elements. To Sanchez-Sepulveda et al. (2019) crucial point regarding interaction was “to improve the interaction with space and objects, the quality of graphics and the reality of the objects” (pp. 379-380) although participants generally had good experiences with movement in the virtual world.

Technical challenges and undeveloped practices of XR were somewhat liminal. Haavik (2016, p.175) brought up the “Janus face of AR”, referring to AR's possibility to enhance but equally to mislead surgical operations from a desired outcome: they noted that doctors and other staff in operation room seemed more concerned about the representations of AR display rather than patients themselves. With their analysis based on the concept of “sensework”, Haavik calls for adequate design and practices for distribution of work. Similarly, Barreau et al. (2015) noted that when it comes to scientific accuracy, scholars such as themselves must be careful with the conclusions they draw from the cases of utilization of XR models.

Lack of resources, such as time and money, hinder both long-term applications and longitudinal research of XR. Although XR may enable multiple scenarios otherwise impossible to achieve, the development of those specific scenarios with XR was an ongoing process in many cases. Occasionally, the emphasis on experimentalism may have led to a hasty application of XR solutions without adequate understanding of situational affordances and hindrances of the applied technology. However, in the article of Barreau et al. (2015), resources for realizing a VR model seemed exceptionally large and resulted in a mouldable, flexible virtual world experience accessible through multiple devices and utilized in various occasions. This meant that they created a full “VR experience” (Sherman and Craig, 2019, pp.736), which in return could and did provide valuable and immersive experience with historical information for historians and other users of the model.

4.2 Employed extended reality applications

4.2.1 Types of employed extended reality mediums

From the 26 articles, 13 handled VR, 7 AR, 4 MR and one both VR and AR. If MR is considered part of AR or vice versa, 13 of the articles handled VR and 11 AR, and one both. AR was more common in training, as it allows interactions with the physical world and is less restrictive regarding traditional communication means. MR was applied in urban planning (Wang and Dunston, 2011), a specific urban planning set up called MRTent (Bratteteig and Wagner, 2012; Wagner, 2012) and regarding Ergomix, a simulation tool used to design an efficient and ergonomic workstation (Hallbeck et al., 2010). However, VR was the common form of XR in the articles qualified for this review.

4.2.2 Immersion and extended reality displays

In the analysis, we make a following division between immersive and non-immersive XR in the real-life settings. Whether accessible via a headset or a desktop computer, if a whole virtual world is available (Sherman and Craig, 2019), XR is considered as immersive. If content covers only narrow, separate objects without offering full spatial experience, even if mixed with the physical world and accessed through a headset, the form of XR is non-immersive. With these prerequisites, the overwhelming majority of included articles utilized immersive virtual worlds rather than mere virtual objects. 16 articles represented immersive XR technology and 10 non-immersive. However, the line between a virtual object and a virtual world should be clarified. Some articles represented either vast environments (e.g. (Dembski et al., 2020; Oprean et al., 2011; Passos et al., 2017)) or a singular restricted area such as a building (Maftei and Harty, 2015), a shelter (Albadra et al., 2020) or a boat (Barreau et al., 2015). These were counted as immersive, since they offer a full spatial experience, opposed to a narrower object, such as a human body in the work of surgeons (Haavik, 2016) or a virtual model of a foot for shoemakers (Mandolini et al., 2017).

Headsets or head-mounted displays (HMDs) were in use in fifteen cases, whereas in six cases virtual objects and worlds were accessible only through computer screens. Wang and Chen (2009) also used data gloves besides of a computing unit and vision-based tracker. Occasionally, more traditional desktop-based means were the most useful and purposeful, such as in the article of Mandolini et al. (2017). The case focused on 3D models of feet for shoes' customization process: seemingly, a large field of view would not bring any additional value. In one case of AR-based training, concrete utilized AR devices were not specified (Anastassova and Burkhardt, 2009). Maftei and Harty (2015) inquired a design case, where design models could be perceived through entire rooms devoted to virtual models, i.e. CAVEs. Another practical XR set-up was MRTent (Bratteteig and Wagner, 2012; Wagner, 2012) located usually nearby a planned site, and the set up consisted of a physical table with superimposed virtual objects as well as multiple screens representing different things on the planned urban site.

Barreau et al. (2015) represented an exceptionally flexible XR case. They described how an immersive sailing experience in a historical ship was constructed, and the result was accessible through multiple devices and displays, such as desktop computers, headsets, mobile phones or as a CAVE environment. Immersion in relation to XR technologies is not only difficult to define as a concept. In addition, ways of realization of immersion in practice are multiple. CAVE is immersive in the sense it gives an idea of whole virtual environment, which can be shared with other users, and without losing the possibility to see one's own body and other possible participants. This is an area of future development of XR, as many other XR solutions do not allow direct and smooth communication between participants.

4.2.3 Interactivity

There are many ways of conceptualizing interactivity. Instead of synthetizing these disparate conceptualizations, we will make the differences visible. Some of the articles referred to interactivity as a feature of technology, i.e. real-time interaction with virtual objects, others saw it as action between human participants whether with or without the aid of extended reality medium. In some cases, interactivity was not regarded at all, as XR was treated as a mere medium to access material resources, such as digitally projected information without additional need to modify or ‘interact’ with them (Anastassova and Burkhardt, 2009).

When regarded as a feature of technology, that is, interaction between the virtual world and the user, interaction or interactivity was used to refer to a capability to customize the content of XR in real time (Barreau et al., 2015; Mandolini et al., 2017). The required level of customization varied. Oprean et al. (2018, p. 424) stated that interactivity of a VR system directly affects the degree of control which each remote collaborator feels while navigating in a virtual meeting space. They ended up operationalizing interactivity as navigability. Regarding AR and following Azuma (1997), Gül (2018) included to AR “a combination of the real and the virtual by providing 3D real time interactivity” (pp. 109-110).

Focusing on technology, two articles addressed XR applications as interactive systems (Sanchez-Sepulveda et al., 2019; Wang and Dunston, 2011). The concept is similar, although not entirely parallel, to a virtual world (Sherman and Craig, 2019), as it emphasizes interactions between the user and the virtual objects. Sanchez-Sepulveda et al. (2019, p.375) noted that interactive systems, as opposed to static virtual environments, “generate less stress”. This was measured quantitatively among other variables without more particular explanations.

If interaction was emphasized between the users, usually the virtual environment itself was somewhat interactive as well or mixture of real and virtual objects (Bratteteig and Wagner, 2012; Lukosch et al., 2015; Passos et al., 2017). Shen et al. (2013) utilized interaction as a concept of human-human communication and human-machine-communication. Wagner (2012) applied entirely different perspective on interaction, as they present “embodied interaction” in the context of MR application MRTent in urban planning. The concept of embodied interaction has been previously applied especially in the field of computer-supported cooperative work (CSCW), but Wagner specifically applies the term in the extended reality context analysing it with the aid of spatiality, representation, and haptic engagement. In the context of MRTent, Wagner (2012) noted that “interactions are difficult to study in a complex setting such as the MR-Tent, as they are enmeshed with all the other design issues we are investigating” (p.38).

4.3 Research on extended reality

4.3.1 Research areas of extended reality

We were interested in the concepts and methods used in XR research regarding actual collaborative working life settings. In the analysed articles, we identified four research areas regarding XR: collaboration, work practices, and evaluation of knowledge transfer.

The articles focusing on collaboration (Anastassova and Burkhardt, 2009; Bratteteig and Wagner, 2012; Da Silva and Garcia, 2013; Gül, 2018; Hayashida, 2018; Lukosch et al., 2015; Nilsson et al., 2011; Oprean et al., 2018) had research emphasis specifically focused on the dynamics of collaboration. For instance, Lukosch et al. (2015) studied situational awareness and team cognition in security training and Sanchez-Sepulveda et al. (2019) conducted research on collaboration in urban planning to increase public's motivation.

Four articles focused on the transformation of practices around XR (Bratteteig and Wagner, 2012; Haavik, 2016; Maftei and Harty, 2015; Wagner, 2012). They surveyed the social and material role of XR in different working life settings (design of a new building, surgical work, and urban planning).

Three articles inquired the efficiency of knowledge transfer via XR (Barreau et al., 2015; Bertram et al., 2015; Grabowski and Jankowski, 2015). Barreau et al. (2015) also described the construction of utilized VR model of historical ship, and the latter two were studies in training context.

These research areas somewhat follow the application areas of XR (design, remote collaboration, and training), which might point to the direction of XR specific research from the viewpoint of social sciences. There were varieties in the emphasis of the studies. 19 articles inquired situations where collaboration was local in one physical space and 7 specified on situations were an extended reality application enabled remote collaboration across different physical locations.

4.3.2 Theoretical and conceptual frameworks to study XR

Authors of the analysed articles applied a variety of theoretical approaches and concepts. Generally, theories and concepts on collaboration and cognition were common, such as “community of practice” applied by Anastassova and Burkhardt (2009) or “situational awareness” by Oprean et al. (2018). When the research focus was on the system development, theories on collaborative systems were often applied (e.g. (Du et al., 2018; Mandolini et al., 2017)).

Schön (1983)'s conceptualizations of reflective practice were applied in multiple studies (Bratteteig and Wagner, 2012; Gül, 2018; Lukosch et al., 2015, Maftei and Harty, 2015). Wagner (2012) did not apply Schön in individually written article although it was utilized in the shared work of Bratteteig and Wagner (2012). Schön was also applied in the articles inquiring collaboration (Lukosch et al., 2015) and work practices (Maftei and Harty, 2015). In their investigation of design practices in construction of a new hospital building, Maftei and Harty (2015) concluded in line with Schön's theory that cave automatic virtual environment (CAVE) worked not only as a mediating knowledge but “bounded up in performing design work configured as social, collaborative and context dependent process” (pp. 72-73).

In addition to Schön, Gül (2018) employed theories on design process and gestures, and especially Gibson (1977)'s theory on affordances. Oprean et al. (2018) utilized it too, but only from secondary sources, whereas Wagner (2012) applied Gibson to form a conceptual framework on embodied interaction and multimodality of XR.

Haavik (2016) utilized their own concept of “sensework” as separated from craftsmanship, describing it as “a label for a type of sociotechnical work in safety-critical operations where groups of professionals try to put together pieces of digital sensor data and different sorts of representations to create a coherent picture that gives meaning to familiar and unfamiliar situations” (p.177). Building, among others, on Latour (2005)’s work, they raised two themes over others: “(1) the role of representations in surgical work and (2) the structure of work and the division of labour” (Haavik, 2016, p.185).

Sensework is not to be mixed with Hayashida’s 2018 “sensecomputing, which in turn investigated how information and communication technology can respond to people’s need for relatedness. Hayashida (2018) from Fujitsu Laboratories relied on behavioural economics and equilibrium theory of intimacy while trying to find solutions to fulfil people’s needs for relatedness in remote work. Extended reality might be a solution for increasing people’s need for relatedness, but as the entire concept and its key features are still maturing, we might ask, following Haavik (2016), what do we actually want or need the technological solutions for.

4.3.3 Methodological frameworks and methods

Case studies represented majority of the research. All research was conducted at least partially as a case study (Yin, 2018), even though not all articles explicitly stated case study as their method. Mainly this was due to our article selection criteria. Presumably, it is difficult to conduct practically oriented research with only interview forms or surveys, although surveys were frequently used to collect additional data (Du et al., 2018; Lukosch et al., 2015; Nykänen et al., 2020; Oprean et al., 2016; Shen et al., 2013). Occasionally, the article included multiple clearly defined cases: Hallbeck et al. (2010) represented five cases, where the same XR tool Ergomix was applied, whereas Bratteteig and Wagner (2012) and Wagner 2012) utilized and inquired MRTent in three different occasions of urban planning. Aldabra et al. (2020) conducted a participatory design research with two forms of workshops, of which one form (“adapt-a-design”) utilised VR.

In some articles, an explicitly stated theory guided methodological choices. For instance, this was the case in Maftei and Harty’s (2015, p.60) article, where Schön’s theory of reflective practice directed the choice of analysing design as practice. In other cases, the connection between methodology and theory was not so specific, such as in Gül (2018)’s article on affordances of AR, where Gül used protocol analysis method (PA) with a theory informed coding scheme.

Other articles addressed the developing methodology for XR research. In addition to their main research aim, four articles from all 23 proposed a new design framework (Shen et al., 2013; Wang and Dunston, 2011; Wang and Chen, 2009) for evaluating the adequacy of XR case by case in different design scenarios or in remote collaboration of security services (Nilsson et al., 2011).

There were few comparative studies with a small sample size, such as a study with three groups of eight people (Bertram et al., 2015) or 21 people (Grabowski and Jankowski, 2015) utilizing quantitative methods. These studies noted the small size of the sample but stated an overall positive result on the use of XR based on quantitative analysis. Bertram et al. (2015), Da Silva and Garcia (2013), and Narasimha et al. (2019) used experimental research design and had control groups for testing different variables, e.g. fastness in a card-sorting task (Narasimha et al., 2019).

Audio and video recordings were the most common way of collecting research material. Whereas interviews were recorded with audio only, video recordings were used for example in Anastassova and Burkhardt's (2009) two-phased study. They first formed a description of automotive technicians training as network of actors based on the concept of community of practice (CoP). In the second study, they observed seven training sessions and gained further insights in CoP “in real training settings” (Anastassova and Burkhardt 2009, p. 717). Video recordings were flexible way to record changing situations in certain space and time: teamwork (Bertram et al., 2015), collaborative urban planning (Bratteteig and Wagner, 2012; Wagner, 2012) or design process of a hospital (Maftei and Harty, 2015). In one case (Haavik, 2016, p.181), the notes were hand-written in an observation room of surgical operations since it was regarded as the “most appropriate”.

As a XR specific data, Dembski et al. (2020) recorded the participants movement in the virtual environment and used it as a research data comparing it to the movement potential of the actual real-life street. This, in its part, helps to estimate effects of new investments in public spaces. Movement recordings were still only a part of the research of Dembski et al. (2020), where a digital twin was used in citizen participation. The research case represented a collaborative design effort of multitude of participants, and equally, virtual reality can be regarded as a part of larger organization of street, building and environmental data collected together.

Although the articles focused on application of technologies, they rarely utilized research methods in natural settings such as ethnography. Bratteteig and Wagner (2012) and Wagner (2012) were the only ones referring explicitly to ethnography regarding their own research. They described their study as an intervention (conducted by the researchers) on three urban planning projects and collected research material through an ethnographic fieldwork. Nevertheless, some ethnographic features existed in the articles in a form of field observations and notes (e.g. (Haavik, 2016)). In addition, Maftei and Harty (2015) briefly referred to ethnomethodology but emphasized simple observation over active participation.

5 Discussion

In this paper we are reviewing research on application areas and collaborative uses of XR in real-life working setting. We have analysed the application areas of XR as well as the added value and limitations of XR technology in practice. One result of the review is that there are still few cases of actual long-term of XR technologies. Design, remote collaboration, and training came forward as the main application areas within the scope of the review. XR as a medium had potential of overcoming obstacles set by time and space, safety, and resources, especially aiding work requiring shared spatial experience. However, XR applications are rather case dependent, and we did not find yet any easily applicable technology that would not require special technical expertise on the matter.

XR solutions are not universal and should be considered case-by-case. For instance, there were social, technological, situational, and practical needs in team training differing from individual training scenarios. If these prerequisites are taken into notice, XR might significantly improve already existing practices, as there were promising results in individual research articles. From the viewpoint of social sciences, one of the strengths of this developing technology lies in enhancing visual and spatial experience, even via more traditional devices such as computer screens. The included articles were written before the COVID-19 pandemic, so the forced remote work might have induced both novel ways of utilization of XR in working life especially in remote collaboration, as well as novel ways of researching XR.

In addition, we were interested in the actual applications and devices utilized as well as their characteristics. There are no established hierarchical relations of varying XR concepts nor established definitions of key features, such as immersion and interaction. We consider this as a sign of “a technology in the making”. Regardless of XR form (VR, AR or MR), there were different levels of immersion and interaction, either with the virtual world and its contents or between participants. Headsets were widely used but equally desktop and screen-based solutions were common: occasionally, the most straightforward and simple technological solutions were the most workable. In a way, we may consider the most powerful technology to be the one that does not draw too much attention towards itself, which supports Weiser’s idea of ubiquitous computing (Weiser, 1991; Weiser and Brown, 1997).

Lastly, we wanted to map theories, key concepts, methodological approaches, and specific research methods around XR. We identified three different research areas: collaboration, evaluation of knowledge transfer, and work practices. Within those areas, theories on collaboration, social interaction and practices were common. Applied research methods were rather traditional and included interviews and observations. Besides these, questionnaires were usually applied as a complementary means.

We expected finding also novel ways of collecting research data in these articles, such as the use of a united screen and recordings of the virtual content. However, in this sense, the results were narrow. More comprehensive data gathering than mere audio and video recording can be considered invasive regarding privacy and self-determination of research participants, thus raising up new concerns on research ethics. In some cases, the virtual world (e.g. a virtual hospital building) was a part of research material (e.g. (Barreau et al., 2015; Maftei and Harty, 2015; Passos et al., 2017)), but the studies did not involve new ways of recording the use of the virtual world. Screenshots were separately mentioned and occasionally also shown in few articles to record time and space specific uses of XR (Bertram et al. ,2015; Du et al., 2018; Gül, 2018; Lukosch et al., 2015; Mandolini et al., 2017; Nilsson et al., 2011; Wang and Dunston, 2011; Wagner, 2012). Further studies on XR should address the development of novel ways of data collection and their overall need and reasonability.

It was also bit surprising how sparsely ethnography was used, as it can potentially contribute to understanding of the sociality and materiality of work in the actual working settings (see e.g. (Carlile et al., 2013)). Means of ethnography may also aid in exploring how the notion of the ‘field site' as a construct provides new ways of conceptualizing ‘work' that extends beyond the workplace (see e.g. (Caliandro, 2014; Leonardi, 2017)).

Our review had some limitations. The sample search was conducted by the first author, which narrows the possibilities of error checking. Second, the final keywords ending “reality” might have excluded relevant articles that handled the same concept with different terms, such as virtual environments (VEs) and cave automatic virtual environments (CAVEs). Maftei and Harty's (2015) article is a good example, which was included in the review as an extra resource. Moreover, the third keyword set (work* OR organization* OR organisation* OR occupation*) may have been somewhat limiting. While forming the final search criteria, we experimented with different keywords before settling to these. There was a tension between wanting as wide and an accurate range of results from different occupations as possible but on the other hand limiting results so that the amount of the search results would have been reasonable. Regarding the screening and eligibility, researchers with more altering backgrounds could have conducted different sample search decisions. Third, occasionally, it was challenging to draw the line between experiments and actual uses of XR in the cases. As noted, we did include some experimental cases in this review, as there was not yet too much research on XR applications in everyday working life use. For instance, Morschheuser et al. (2017) researched cooperation in games, where they inquired the relationship between game features and we-intentions in an augmented reality game. The article was excluded due to its loose connection to working life, but it is possible that application of XR in looser contexts than strict working life projects might lead to most serviceable solutions.

It can be difficult to separate whether the research is technology driven. We tried to solve this by searching especially for articles where the technology was not necessarily in the main role. Rather, it might have worked as means for solving or aiding something, as oppose to scenarios particularly created for technology testing. This helped us to look for as close real-life settings as possible in the articles. Eventually, this approach partially caused the exclusion of significant amount of research in design area.

6 Conclusion

Overall research results from the analysed articles indicate the potential of XR. Despite our original goal of reviewing more established and stabilized uses of XR, long-term cases were rare. The short-term nature of XR utilization may explain partially the lack of ethnographic and longitudinal approaches. The relation between a long-term application and longitudinal research can be seen as a sort of a chicken or an egg dilemma: how to apply XR in long-term perspective without adequate research on its supportive implementation practices? How to understand what are the best practices without examples of stabilized uses? These limitations and open questions around XR further emphasize the need for development of adequate practices and research on them.

Increasing availability and intuitiveness of XR compares with somewhat similar trends of already embedded information and communication technology, such as mobile phones, laptops, and wireless headphones. For now, XR still holds significant potential rather than clearly confirmed general advantages in working life. The applicability of XR must be addressed case by case taking notice the specific affordances of each piece of technology for needs in a current situation. This requires a multidisciplinary perspective, where technologically oriented solutions can be developed and challenged using research perspectives from social and human sciences. This will help in constructing a better understanding of potential XR applications areas. More research from the perspective of social sciences on uses of XR as a part of concrete working practices is needed to get a better understanding of prerequisites and consequences of the use of XR technologies in real working life settings.

Notes

  • In addition, the first author consulted the information specialist of the Helsinki University Library during fall 2019 and conducted multiple experimental searches, which supported the selection of the three search engines.
  • Two separate searches were conducted due to delay in prolonged article preparation and review process.
  • After additional selection of subject categories in Web of Science, the amount of 313 articles decreased to 213. The excluded subject categories can be found in the footnote of the appendix 1. During the 2020 search the records of excluded subject categories can be accessed through the URL-link provided in the table of appendix 2 This change is due to Clarivate Analytics development of Web of Science search.
  • The full records of the conducted search with three different academic search engines in detail can be found as appendixes at the end (Appendix 1. And 2. Search engine specifications) to enable evaluation, transparency, repetition and further modification of the search for varying purposes.
  • Separate Prisma Diagrams of searches 1 and 2 can be provided.

References

  • Adriaanse LS, Rensleigh C (2013) Web of Science, Scopus and Google Scholar: A content comprehensiveness comparison. The Electronic Library, 31(6), pp 727-744 doi:10.1108/el-12-2011-0174
  • Albadra D, Elamin Z, Adeyeye K, Polychronaki E, Coley DA, Holley J, Copping A (2020). Participatory design in refugee camps: comparison of different methods and visualization tools. Building Research and Information, 49(2), 248-264. https://doi:10.1080/09613218.2020.1740578
  • Anastassova M, Burkhardt J (2009) Automotive technicians’ training as a community-of-practice: Implications for the design of an augmented reality teaching aid. Applied Ergonomics, 40(4), pp 713-721 doi:10.1016/j.apergo.2008.06.008
  • Arksey H, O’Malley L (2005). Scoping studies: Towards a methodological framework. International Journal of Social Research Methodology, 8(1), 19-32. doi:10.1080/1364557032000119616
  • Aukstakalnis S (2017) Practical Augmented Reality: A guide to the technologies, applications, and human factors for AR and VR. Addison-Wesley Professional.
  • Azuma, RT (1997) A Survey of Augmented Reality. Presence: Teleoperators & Virtual Environments, 6(4), pp 355-385 doi:10.1162/pres.1997.6.4.355
  • Barreau J, Nouviale F, Gaugne R, Bernard Y, Llinares S, Gouranton V (2015) An immersive virtual sailing on the 18th-century ship Le Boullongne. Presence: Teleoperators and Virtual Environments, 24(3), pp 201-219 doi:10.1162/pres_a_00231
  • Berg L, Vance J (2017) Industry use of virtual reality in product design and manufacturing: A survey. Virtual Reality, 21(1), pp 1-17 doi:10.1007/s10055-016-0293-9
  • Bertram J, Moskaliuk J, Cress U (2015) Virtual training: Making reality work? Computers in Human Behavior, 43(C), pp 284-292 doi:10.1016/j.chb.2014.10.032
  • Booth A, Sutton A, Papaioannou D (2016) Systematic approaches to a successful literature review, 2nd edn. Sage
  • Bratteteig T, Wagner I (2012) Spaces for participatory creativity. CoDesign: Participatory Engagement in Design, 8(2-3), pp 105-126 doi:10.1080/15710882.2012.672576
  • Brown J, Duguid P (2002) The social life of information. Harvard Business School Press
  • Caliandro A (2014) Ethnography in digital spaces: Ethnography of virtual worlds, netnography, and digital ethnography. Handbook of Anthropology in Business, pp 738-761
  • Carlile PR, Nicolini D, Langley A,Tsoukas H (2013) How matter matters: Objects, artifacts, and materiality in organization studies. Oxford.
  • Coole DH, Frost S (2010) New Materialisms: Ontology, Agency, and Politics. Duke University Press
  • da Silva C, Garcia A (2013) A collaborative working environment for small group meetings in Second Life. SpringerPlus, 2(1), pp 1-14 doi:10.1186/2193-1801-2-281
  • Davids J, Manivannan S, Darzi A, Giannarou S, Ashrafian H, Marcus HJ (2020) Simulation for skills training in neurosurgery: a systematic review, meta-analysis, and analysis of progressive scholarly acceptance. Neurosurgical Review, 1-15. doi:10.1007/s10143-020-01378-0
  • Dawidowicz P (2010) Literature reviews made easy: A quick guide to success. Charlotte, NC: Information Age Pub.
  • Dembski F, Wossner U, Letzgus M, Ruddat M, Yamu C (2020). Urban Digital Twins for Smart Cities and Citizens: The Case Study of Herrenberg, Germany. Sustainability, 12(6). https://doi:10.3390/su12062307
  • Du J, Shi Y, Zou Z, Zhao D (2018) CoVR: Cloud-Based Multiuser Virtual Reality Headset System for Project Communication of Remote Users. Journal of Construction Engineering and Management, 144(2). doi:10.1061/(ASCE)CO.1943-7862.0001426
  • Fast-Berglund Å, Gong L, Li D (2018) Testing and validating Extended Reality (xR) technologies in manufacturing. Procedia Manufacturing, 25, pp 31-38 doi:10.1016/j.promfg.2018.06.054
  • Fox J, Arena D, Bailenson JN (2009) Virtual reality: A survival guide for the social scientist. Journal of Media Psychology, 21(3), pp 95-113
  • Gibson J (1977) The theory of affordances. In: Shaw R, Bransford J (eds) Perceiving, acting and knowing: Toward an ecological psychology. Erlbaum Associates, Hillsdale, NJ, pp 67–82
  • Grabowski A, Jankowski J (2015) Virtual Reality-based pilot training for underground coal miners. Safety Science, 72, pp 310-314 doi:10.1016/j.ssci.2014.09.017
  • Gül L (2018) Studying gesture-based interaction on a mobile augmented reality application for co-design activity. Journal on Multimodal User Interfaces, 12(2), pp 109-124 doi:10.1007/s12193-017-0252-0
  • Haavik T (2016) Keep your coats on: Augmented reality and sensework in surgery and surgical telemedicine. Cognition, Technology & Work, 18(1), pp 175-191 doi:10.1007/s10111-015-0353-z
  • Haddaway NR, Collins AM, Coughlin D, Kirk S (2015) The role of Google Scholar in evidence reviews and its applicability to grey literature searching. PloS one, 10(9), e0138237 doi:10.1371/journal.pone.0138237
  • Hallbeck MS, Bosch T, Van Rhijn G, Krause F, De Looze MP, Vink P (2010) A tool for early workstation design for small and medium enterprises evaluated in five cases. Human Factors and Ergonomics in Manufacturing & Service Industries, 20(4), pp 300-315 doi:10.1002/hfm.20222
  • Hayashida N (2018) Sensecomputing for human-machine collaboration through human emotion understanding. Fujitsu Scientific and Technical Journal, 54(5), pp 62-69
  • Jamei E, Mortimer M, Seyedmahmoudian M, Horan B, Stojcevski A. (2017) Investigating the role of virtual reality in planning for sustainable smart cities. Sustainability, 9(11) doi:10.3390/su9112006
  • Kokkonen A, Alin P (2015) Practice-based learning in construction projects: A literature review. Construction Management and Economics, 33(7), pp 1-18 doi:10.1080/01446193.2015.1062903
  • Latour B (2005) Reassembling the social: An Introduction to Actor-Network-Theory. Oxford, OUP Oxford
  • Leonardi PM (2017) Methodological guidelines for the study of materiality and affordances. In: Mir R, Jain S (eds) Routledge companion to qualitative research in organization studies. Routledge, pp 279-290 doi:10.4324/9781315686103-18
  • Li X, Yi W, Chi H, Wang X, Chan AP (2018) A critical review of virtual and augmented reality (VR/AR) applications in construction safety. Automation in Construction, 86, pp 150-162 doi:10.1016/j.autcon.2017.11.003
  • Lukosch S, Lukosch H, Datcu D, Cidota M (2015) Providing Information on the Spot: Using Augmented Reality for Situational Awareness in the Security Domain. Computer Supported Cooperative Work (CSCW), 24(6) doi:10.1007/s10606-015-9235-4
  • Madary M, Metzinger TK (2016). Real virtuality: a code of ethical conduct. Recommendations for good scientific practice and the consumers of VR-technology. Frontiers in Robotics and AI, 3, 3.
  • Maftei L, Harty C (2015) Designing in caves: Using immersive visualisations in design practice. ArchNet-IJAR: International Journal of Architectural Research, 9(3), pp 53-75 doi:10.26687/archnet-ijar.v9i3.693
  • Mandolini M, Brunzini A, Germani M (2017) A collaborative web-based platform for the prescription of Custom-Made Insoles. Advanced Engineering Informatics, 33, pp 360-373 doi:10.1016/j.aei.2016.10.004
  • Mann S, Havens JC, Iorio J, Yuan Y, Furness T (2018). All reality: values, taxonomy, and continuum, for virtual, augmented, eXtended/MiXed (X), Mediated (X, Y), and multimediated reality/intelligence. Paper presented at the Presented at the AWE 2018.
  • Miettinen R, Paavola S (2014) Beyond the BIM utopia: Approaches to the development and implementation of building information modeling. Automation in Construction, 43, pp 84-91 doi:10.1016/j.autcon.2014.03.009
  • Mikropoulos TA, Natsis A (2011) Educational virtual environments: A ten-year review of empirical research (1999–2009). Computers & Education, 56(3), pp 769-780 doi:10.1016/j.compedu.2010.10.020
  • Milgram P, Kishino F (1994) A taxonomy of mixed reality visual displays. IEICE Transactions on Information and Systems, 77(12), pp 1321-1329
  • Moher D, Liberati A, Tetzlaff J, Altman DG, The PRISMA Group (2009) Preferred Reporting Items for Systematic Reviews and Meta-Analyses: The PRISMA Statement. PLoS Med 6(6): e1000097. doi:10.1371/journal.pmed1000097
  • Moher D, Stewart L, Shekelle P (2015) All in the family: Systematic reviews, rapid reviews, scoping reviews, realist reviews, and more. Systematic Reviews, 4(1) doi:10.1186/s13643-015-0163-7
  • Moore GE (1998) Cramming More Components onto Integrated Circuits. Proceedings of the IEEE, 86(1), pp 82-85 doi:10.1109/JPROC.1998.658762
  • Moore GE (1995) Lithography and the future of Moore's law. Paper presented at the Integrated Circuit Metrology, Inspection, and Process Control IX, 2439 pp 2-17 doi:10.1117/12.209151
  • Narasimha S, Dixon E, Bertrand JW, Chalil Madathil K (2019) An empirical study to investigate the efficacy of collaborative immersive virtual reality systems for designing information architecture of software systems. Applied Ergonomics, 80, pp 175-186 doi:10.1016/j.apergo.2019.05.009
  • Nikolić D, Maftei L, Whyte J (2019) Becoming familiar: how infrastructure engineers begin to use collaborative virtual reality in their interdisciplinary practice. Journal of Information Technology in Construction (24) pp 489–508 doi:10.36680/j.itcon.2019.026
  • Nilsson S, Johansson BJE, Jonsson A (2011) Cross-Organizational Collaboration Supported by Augmented Reality. IEEE Transactions on Visualization and Computer Graphics, 17(10), pp 1380-1392 doi:10.1109/TVCG.2010.249
  • Nykänen M, Puro V, Tiikkaja M, Kannisto H, Lantto E, Simpura F, . . . Teperi AM (2020). Implementing and evaluating novel safety training methods for construction sector workers: Results of a randomized controlled trial. Journal of Safety Research. http://doi:10.1016/j.jsr.2020.09.015
  • Oprean D, Simpson M, Klippel A (2018) Collaborating remotely: An evaluation of immersive capabilities on spatial experiences and team membership. International Journal of Digital Earth: Virtual Geographic Environment for advanced geographic analysis, 11(4), pp 420-436 doi:10.1080/17538947.2017.1381191
  • Orlikowski WJ (2007) Sociomaterial practices: Exploring technology at work. Organization studies, 28(9), pp 1435-1448. https://doi:10.1177/0170840607081138
  • Pan X, Hamilton A (2018) Why and how to use virtual reality to study human social interaction: The challenges of exploring a new research landscape. British Journal of Psychology, 109(3), pp 395-417.
  • Paez A (2017) Gray literature: An important resource in systematic reviews. Journal of Evidence‐Based Medicine, 10(3), pp 233-240 doi:10.1111/jebm.12266
  • Passos C, Silva M, Mol A, Carvalho P (2017) Design of a collaborative virtual environment for training security agents in big events. Cognition, Technology & Work, 19(2-3), pp 315-328 doi:10.1007/s10111-017-0407-5
  • Radianti J, Majchrzak T, Fromm J, Wohlgenannt I (2020) A systematic review of immersive virtual reality applications for higher education: Design elements, lessons learned, and research agenda. Computers and Education, 147, 103778–. doi:10.1016/j.compedu.2019.103778
  • Riva G (1999) Virtual Reality as Communication Tool: A Socio-cognitive Analysis. Presence, 8(4), pp 462-468 doi:10.1162/105474699566341
  • Sanchez-Sepulveda M, Fonseca D, Franquesa J, Redondo E (2019) Virtual interactive innovations applied for digital urban transformations. Mixed approach. Future Generation Computer Systems, 91, pp 371-381 doi:10.1016/j.future.2018.08.016
  • Schön D (1938) The reflective practitioner. How professionals think in action. London, Temple Smith
  • Seymour M, Riemer K, Kay J (2018) Actors, avatars and agents: potentials and implications of natural face technology for the creation of realistic visual presence. Journal of the Association for Information Systems, 19(10), 4 Retrieved from https://aisel.aisnet.org/jais/vol19/iss10/4
  • Shen W, Zhang X, Qiping Shen G, Fernando T (2013) The User Pre-Occupancy Evaluation Method in designer–client communication in early design stage: A case study. Automation in Construction, 32, pp 112-124 doi:10.1016/j.autcon.2013.01.014
  • Sherman WR, Craig AB (2019) Understanding virtual reality: Interface, application, and design, 2nd edn. Morgan Kaufmann, San Francisco, CA
  • Slater M (2009). Place illusion and plausibility can lead to realistic behaviour in immersive virtual environments. Philosophical Transactions of the Royal Society of London B: Biological Sciences, 364(1535), 3549–3557. Retrieved from http://rstb.royalsocietypublishing.org/content/364/1535/3549.short
  • Slater M., Wilbur S (1997). A framework for immersive virtual environments (FIVE): Speculations on the role of presence in virtual environments. Presence: Teleoperators & Virtual Environments, 6(6), pp 603-616.
  • Stanton NA, Plant KL, Roberts A P, Allison CK, Howell M (2020) Seeing through the mist: an evaluation of an iteratively designed head-up display, using a simulated degraded visual environment, to facilitate rotary-wing pilot situation awareness and workload. Cognition, Technology & Work, 22(3), pp 549-563 doi:10.1007/s10111-019-00591-2
  • Van Wee B, Banister D (2016) How to write a literature review paper? Transport Reviews, 36(2), pp 278-288 doi:10.1080/01441647.2015.1065456
  • Wagner I (2012) Building Urban Narratives: Collaborative Site-Seeing and Envisioning in the MR Tent. Computer Supported Cooperative Work (CSCW), 21(1), pp 1-42 doi:10.1007/s10606-011-9152-0
  • Wang X, Chen R (2009) An experimental study on collaborative effectiveness of augmented reality potentials in urban design. CoDesign, 5(4), pp 229-244 doi:10.1080/15710880903320020
  • Wang X, Dunston, PS (2011) Comparative Effectiveness of Mixed Reality-Based Virtual Environments in Collaborative Design. IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews), 41(3), pp 284-296 doi:10.1109/TSMCC.2010.2093573
  • Wang P, Wu P, Wang J, Chi H, Wang X (2018) A Critical Review of the Use of Virtual Reality in Construction Engineering Education and Training. International Journal of Environmental Research and Public Health, 15(6). doi:10.3390/ijerph15061204
  • Weiser M (1991) The computer for the 21st century. Scientific American, 265(3), pp 94-105 doi:10.1145/329124.329126
  • Weiser M, Brown JS (1997) The coming age of calm technology. Beyond calculation. Springer. pp 75-85 doi:10.1007/978-1-4612-0685-9_6
  • Whyte J, Nikolić D (2018) Virtual Reality and the Built Environment. Routledge, London doi:10.1201/9781315618500
  • Yan W, Culp C, Graf R (2011) Integrating BIM and gaming for real-time interactive architectural visualization. Automation in Construction, 20(4), pp 446-458 doi:10.1016/j.autcon.2010.11.013
  • Yin R (2018) Case study research and applications : design and methods, 6th edn. SAGE
  • Zahabi M, Abdul Razak A (2020) Adaptive virtual reality-based training: a systematic literature review and framework. Virtual Reality: the Journal of the Virtual Reality Society https://doi:10.1007/s10055-020-00434-w

Authors


Minna Vasarainen

minna.vasarainen@gmail.com

ORCID iD

Affiliation : PhD student

Country : Finland


Sami Paavola

ORCID iD

Affiliation : Associate Professor

Country : Finland


Liubov Vetoshkina

ORCID iD

Affiliation : Post-doc reseacher

Country : Finland

Attachments

No supporting information for this article

Article statistics

Views: 570

Downloads

PDF: 70

XML: 4

Citations