An Agent-Aware Computing Based Mixed Reality Information System

In this paper, we present an agentaware computing based collaborative information system scheme for MR applications. The HCI issues in VR, AR, and MR, have been discussed firstly. Besides the equipment, the lack of understanding of users also accounts for a significant bottleneck of improving user experience and immersion during the interaction. Aiming at the issues, an agent-aware computing based scheme is proposed and implemented. The corresponding systematic scheme is presented in the views of functionality, modules and system workflow and discussed in design, implementation, and system synthesis. By means of the human-aware computing, system-aware computing and human-system aware computing, issues such as mixed reality fusion, QoE-QoS management, confidentiality, and security, are addressed by the applying agentaware computing based scheme into an implementation of the collaborative information system for mixed reality. Related modules along with their function and test results are given and discussed. Index Terms mixed reality fusion; collaborative information processing; agent-aware computing based design


Introduction
Virtual Reality (VR) refers to an alternate world filled with simulated environments and objects. Computer-generated images allow the end user to interact with the virtual space. The virtual reality presents the user the experience of a mixture of reality and virtuality (Coates 1992;Greenbaum 1992;Steuer 1992). One of the important issues in VR is immersive interaction (Bowman and McMahan 2007;Psotka 1995).
VR can be divided into 3 kinds (see table 1): textual VR, which refers to interaction without visualization; desktop VR, interaction with visualization; immersive VR, interaction with high level of immersion(Burdea Grigore and Coiffet 1994).
Augmented Reality (AR) refers to an experience in a direct or indirect view of a physical, real-world environment.
An AR system provides a virtual layer of information with computer-generated images in reality. Compared to VR, E-mail: charles.liu@mq.edu.au AR consists of more reality than artificially modeled virtuality. Many AR systems are more widely used than immersive VR systems due to their low cost and easily accessible characteristics (Ong and Nee 2013; Lee and Akin 2011; Kaufmann and Schmalstieg 2003). However, when immersive interaction is needed, a mixture of virtual and real worlds supported with equally physical and augmented elements is required.
Mixed reality (MR) systems (Milgram and Kishino 1994; Tamura, Yamamoto, and Katayama 2001; Costanza, Kunz, and Fjeld 2009) were proposed to perform merging of real, with a cyber world. Due to this physical and digital coexistence, a mixed reality system allows individuals to experience a high level of immersion. Reproduced multimedia metaphors (related to senses of sight, hearing, smell, taste and touch), in MR encompass both augmented reality and augmented virtuality. This type of systems provide possible improvements for high-level of interaction, such as memory functioning (Parsons and Rizzo 2008), attention enhancement (Cho et al. 2002) and behavior therapy (Klinger et al. 2005).
To improve the performance of MR systems for a highlevel of interaction, modern computer systems have provided many auxiliary functionalities. These perform analysis on data and visualization with marked characteristics (Menache 2000;Bickel et al. 2007), and require some sensors and actuators (as in (Gulrez and Kavakli 2007)) for such interaction. However, due to the lack of understanding of users' intention, those devices play roles similar to keyboards and mice. The MR systems perform mostly passive interaction. Passive interaction leads to a reactive approach in current HCI practices (Stephanidis 2001), and this reactive interaction may reduce the quality of user experience and the usability of the system, particularly for the scenario that user needs the system to assist in task completion.
The goal of this paper is to develop a framework for an intelligent HCI (iHCI) system using a collaborative information system for MR in (Liu and Kavakli submitted; ) and allow the system to understand users intention during the interaction. This requires the systems to be able to understand the subject and context in human-computer interaction. This will lead to a better user experience and information security using context-aware computing (Hong and Landay 2001;Gay 2009). The context refers to the environment, condi- tions and surroundings that involve in the subject system and its computing. Context-aware computing here refers to computing operations of computers to simulate awareness of context by sensing the physical environment, and adapting systems' behavior accordingly.

Related Works
The applications of computational intelligence in virtual environments can be seen as a convergence of smart system and HCI systems. How to develop an intelligent HCI (iHCI) system is a key to this convergence. Agent-based design can be used to implement the context aware iHCI systems. There are many studies (e.g. (Li 2015;Ruhland et al. 2014;Kinateder et al. 2014; Luck and Aylett 2000; Rickel and Johnson 1999)) on intelligent system design. Issues related to intelligent VR systems, are knowledge representation, autonomous agents, and agent coupling. They showed perform some capabilities for intelligence, such as perception, cognition, recognition and instruction. However, those intelligent interactions were based on the results of machine states influenced by the users' operation. Even though some complex factors (such as social influence) have been considered, the behavior of agents still are controlled by a certain dynamic system model rather than the context.
Networking applications mainly have focused on communication, traffic flow and implementation of the interactive systems (e.g. (Pouliquen-Lardy et al. 2016; Kurup and Liu 2016;Gauglitz et al. 2014)).The applications of computational intelligence have involved interactive elements, such as virtual objects, virtual roles and real user connected by the network. The issue of optimization of traffic flow was also addressed in those works. However, iHCI systems mostly focused on the performance of the system, rather than the performance of user experience.
iHCI systems and other smart systems, e.g., robotics, share similar properties, such as real-time processing, power efficiency, computing resources, which are common issues in implementation. However, intelligent HCI system focus more on the performance of interaction and user experience than a robotic system. Therefore, the user experience is an integral part of iHCI.
As for the user experience, there are many studies conducted to explore the affects on users during the interaction (e.g. . The factors, such as graphic displays, dialog design, and operation interface were studied. Those works provide references to improve the user-experience in HCI. However, an iHCI system must perform a realtime user experience assessment. It is a challenge for a machine to understand abstract concepts like user experience. Agent-based models (ABM (Getchell 2008;Helbing 2012)) are micro-scale models that simulate the simultaneous operations and interactions of multiple agents to create or predict complex phenomenon. Simple behaviorial rules generate complex behavior. Thus, agents may experience learning and adaptation. Knowledge systems can help HCI to acquire human sense of interaction and experience. Many works related to knowledge-based HCI focused on knowledge translation and knowledge system im- The knowledge in these works is mainly used as a logic of rules and data. Efficacy of the knowledge-based instruction was the primary criteria of evaluation. However, an iHCI system must take the human perception and cognition into account to simulate the real response by a human rather than a logic link between the rules and the knowledge base. Only with this process can human experience be modeled and computed. However, human perception is abstract. One of its essential characteristics is uncertainty. Subjective awareness is hard to quantify, while computation needs to be specific and accurate for the processing of interactive data. How to solve this contradiction is a challenge for intelligent HCI system.
Moreover, when considering wide area mixed reality applications that run in an open field of interaction (Makita et al. 2016; You and Thompson 2017), the system must be the aware of its own operations. This awareness is particularly useful for information security and privacy management. due to the possibility of public access in an open architecture. Since information of users' behavior and response will be generated during immersive interaction, this type information may be sensitive as well, in terms of privacy, especially in therapy, education, and military (Elliott, de Bruin, and Dumoulin 2015; Merchant et al. 2014;Rizzo et al. 2011). In some cases, some critical data such as photo, video and body measurements may be involved as well. When a spying node is placed in the system, the profiles of the users would be stolen, and the system itself would be attacked with malware (e.g. man in the middle attack malware). If the system had self-awareness to detect such anomalousness, it can detect the unusual changes and protect the system as well as users' privacy. Therefore, it can be seen that the awareness enables the HCI systems to perform advance interaction with humanized experience and the information security.
To perform such awareness, a system may need sensors to collect the information about the users and itself. This design implies that the system is intricate with considerable the amount of complexity requirement for processing. Consequently, the cost for establishing the cyber environment can be cut down, due to the advantages brought by the sensors.
The issues above motivates us to develop a mixed reality system for enhanced awareness. Based on the analysis above, we explore solutions to enable an HCI system to perform smart interaction with agent aware computing enhanced intelligence, and achieve a vision based scheme for the mixed reality system, thus providing support for the implementations of vision based smart interactive systems for mixed reality applications.

Agent-Aware Computing
The concept of Agent Aware Computing (AAC) refers to computational processing for enabling the machine to be aware of an agent operating in the system, including human, devices and the combination of the human and devices. Such systems can analyze the features, and attributes of an agent using an awareness model. The targets and subjects involved in the interaction with the data captured as the result of the analysis will be stored as knowledge for processing to complete a certain task. Corresponding MR system using the agent aware computing to enhance its functionalities is named as an Enhanced Mixed Reality System, abbreviated as enhanced mixed reality (eMR) system.

Methodological Functions
In terms of enhancing the interaction in a mixed reality system (i.e., enhanced mixed reality, eMR), as shown in Figure 1, we divide the tasks of agent aware computing into three main groups: human aware, system aware, and humansystem aware; 1. human-aware computing (HAC) targets to simulate human visual attention, and memory. The main tasks of HAC include: simulating attention to the target a) to distinguish the target in the scene and b) to differentiate it from the other object as a background, and memory fusion, i.e., to fuse the target of interest with another scene; 2. system-aware computing (SAC) targets to make the system aware of its devices, network, and operations related to the performance and processing. The main task of SAC include a) performing the authentication to the devices to detect the well-intended and malicious operations; b) being aware of the structure and operations of the system for the processing, such as knowledge learning and updating data; 3. human-system-aware computing (HSAC) targets to make the system aware of both sides of human and system in processing. The main tasks of HSAC is to perform the comprehension of both the user experience and system performance.
The methodological functions to support the agent-aware computing scheme, include: 1) target of interest (ToI)-aware computing: to make the system aware of the subject of interaction as a target of interest; 2) context-aware computing: to Figure 1: Agent-Aware Computing Functional Schema make the system aware of the set of circumstances or facts during the interaction and processing; 3) user-experience aware computing: to enable the system to be aware of the experience of a user during the interaction. The context in context-aware computing consists of two orientations: 1) context of interaction: the context of the scenario related to the perception of real world that surrounds the subject in a particular event, situation, etc.; 2) context of the system environment: the context of the cyber-environment related to the devices, operations, and processes to perform the functionalities of the system.

Implementation
To simulate human visual attention and memory, we propose to use visual computing as visual computing is closer to the way how humans perceive. It also alleviates the constraints on the implementation, which suits the goals of humanaware computing well. We apply the recursive learning and fuzzy adaptive learning strategies to form awareness gradually (Liu and Kavakli 2016b). The target of interest recognition is performed by a recursive α optimization. Corresponding knowledge is stored as a fuzzy set with learning parameters.
To implement target awareness, we apply a non-parameter stochastic adaptive learning strategy ). Based on Gaussian Modeling (GM), we have developed a kernel learning strategy, which is designed to improve the adaptivity of the recognition. This enables the system to differentiate the target by motion cues and perform a scenario layering task without the support of prior knowledge of the scene. In addition, we have designed a scheme to simulate the stereo visual awareness. A context-aware computing strategy to recognize patterns is designed and applied to simulate human awareness and processing of stereo pattern analysis (Liu and Kavakli 2016c). Based on the knowledge based inference system, human experience can be used to model the correlations of the patterns in the context. We have developed an enhanced learning approach to allow the system to process the ambiguous patterns and to refine confidence.
To implement system-awareness (SA), we apply dataaware computing using authentication. With the security enhancement for confidential communication the system can be aware of the authorities of its devices, in the network and detect the potential malicious attacks, using public key cryptography (Liu and Kavakli 2017). This brings system awareness in terms of system security. Regarding operations, we focus on learning and knowledge management to contribute to system-awareness. System-awareness has been modeled as a ξ process (Liu and Kavakli 2016e). A scalable learning strategy is designed to simulate the process of knowledge broadening and applied to perform scalable management using a dispersed structure.
To implement human-system-aware computing (HSAC), we model the comprehension of user experience and system performance based on subspace learning with the quality of experience (QoE) and the quality of service (QoS), and use the model to estimate user experience (Liu and Kavakli 2016a), by sensing the service.

System Design
Human vision is approximately ten times as much as the depth detector. In some cases, the texture of the subjects is more important than the distance, for example, in applications (such as (Shah et )). With visual computing, the system's dependence on the sensors is removed. This brings more flexibility to the implementation. For example, participants would no longer need to wear a suit with many sensors and transmitters to communicate with the receivers in a virtual environment. Responses of the system are delivered according to the visual information captured by other cameras or other types of optical sensing technologies (e.g., infrared sensors). Therefore, interaction would be simplified in terms of the calibration for networking as well as transmitting-receiving test. Meanwhile, the number of the participants will not be limited by the amount of devices or suit. Special effects can be added to the simulation, which brings users more vivid experience during the interaction. Thus, it provides another option for acquiring better visual effect with less device dependence.
There are several issues when implementing a collaborative information system for MR (Liu and Kavakli 2016d), including: scenario awareness and target awareness, knowledge and experience modeling, scalable knowledge management, confidentiality and security.
The design of the system is divided into three main subsystems: Mixed Reality Fusion Subsystem (MRFS), User Awareness Subsystem (UAS) and Sensing Network Subsystem (SNS), as shown in Figure 2. Each subsystems are supported by related modules.
• In the MRFS subsystem, Enhanced Matting System (EMS), Dynamical Scene Layering (DSL), and Context-Aware Computing (CAC) support the system to adapt to the texture, motion and depth data; • In the UAS subsystem, User Identification Module (UIM) supports the system to recognize the user, providing the corresponding information for interaction; • In the SNS subsystem, System Security Module (SSM) is designed for information system to guarantee the information security in collection, organization, storage and communication of information during the HCI interaction.
Quality of Experience (QoE) -Quality of Service (QoS) Management (QoE-QoS-M) module serves as a joint module for both the UAS and SNS. QoE and QoS play an important role for both the user awareness and the network system. With the support of the QoE-QoS-M, the balance between user experience and system performance can be used to improve the usability of the whole system.
In this paper, texture based recognition, motion based recognition and context-aware computing, are studied and implemented to enhance the differentiation. Principles of optimization to reduce the uncertainties are studied and discussed. Corresponding optimization strategies are proposed to guarantee the stability of the solution.
QoE-QoS management is applied to assess the quality for the balance between user experience and communication performance. This module is mainly responsible for the awareness of user experience and system performance. We implemented a data-aware strategy for QoE-QoS management. Three main types of QoE-QoS management (optimal QoE management, optimal QoS management, and QoE-QoS balance management) are discussed. With the consideration of user-experience and system performance, a unified optimization strategy with quantification and normalization of data space has been proposed to perform QoE-QoS balancing.
After the assessment, the data volume is re-encoded in a compression form. A protocol with PKC based cryptography is designed to perform the encryption and decryption to guarantee the confidentiality of the dataflow and authentication, implementing the awareness of confidentiality and agent identification (in terms of machinery) to reinforce the security of the system. With the consideration of both efficiency and performance, a protocol has been implemented to establish a chaotic tunnel for logical peer to peer communication. The system can detect anomalous node in the network and protect the both the information and communication against attacks. Related strategies are designed with the consideration of the key space, sensitivity, correlation and uniform distribution to enhance the self-awareness of the system.
The pattern analysis is applied to process the authenticated data, recognizing the target of interest (TOI) in the captured scene to support stereo modeling and scenario fusion by the awareness of interactive target. A user identification module is applied to the system to implement the agent identification. Face identification using a dispersed knowledge system is used to test the flexibility of the processing. The identification result impacts on all the modules related to user data processing regarding information security and privacy management, including the user scene data,  and the user stereo model database. The detailed implementation is given in (Liu and Kavakli ). A knowledge system with adaptive fuzzy learning strategy for visual differentiation has been designed to cope with the uncertainty in the knowledge as well as processing with a recursive learning strategy for refining the enhancement of the knowledge. A scalable learning strategy has been implemented for dispersed knowledge management. Both data scalability and knowledge usability are considered.

Enhanced Mixed Reality System
The system synthesis is presented to demonstrate how to implement data fusion for a mixed reality application. Each module is designed according to the works (Liu and Kavakli 2016d; 2016b; 2016c; 2016a; 2016e; 2017), and corresponding processing is synthesized as an integrated system. The system is designed under a CIS architecture (as shown in Figure 3), where the agent in the system can be either a human user or machinery (computer, camera, or other devices) or combination (user with a machine). All agents can collect data from and upload to the system depending on their roles. Figure 4 shows the system architecture and the work flow among the components. Vision is used as the information source to overcome the constraints of a sensor based data system. In the system (Figure 4), the input is from imaging systems that provide visual data of target scene (details in Section ). Since high-resolution data will be expensive in terms of computational resource, before transmission, the data is processed by QoE-QoS management system (de-  . This brings balance between user experience and the communication performance, and provides the awareness of user experience and system performance.

Data
Considering information security, we introduced a protocol with PKC based cryptography to perform encryption and decryption to guarantee the confidentiality of the dataflow (details in Section ). In this part, the awareness of confidentiality and agent identification (in terms of machinery) is implemented to reinforce the security of the system. The authorized data moves to pattern analysis (details in Section ) to differentiate the target of interest (TOI) in the captured scene. This supports stereo modeling and scenario fusion (Section ). For privacy management, a user identification module is developed. This implements the agent identification in the view of human (Section ).

Camera System
A camera system is designed for capturing the visual data. To perform stereo modeling, a multiple-view camera system is applied. The scheme is shown in Figure 5, in which two cameras capture the visual information from the user's righthand side profile and frontal view. We project the user's image into a virtual reality platform. (as shown in Figure 6). Figure 7 shows the detailed setup in the laboratory. A user stands at the central point surrounded by camera A and B (as shown in Figure 8 (a)).
The resolution of the cameras is 4032 × 3024. The image is converted into 3840 × 2160 for 4K displays. H.265 compression is applied to reduce the size of the data for commu- , the view of the real world is captured by an outdoor camera system (as shown in Figure 8 (b)). The scene including user and real world is processed as an example. Figure 9 shows the real world scene samples of User A, B, and C, as well as an outdoor scene. Each camera system serves as an independent agent for data collection.

QoE-QoS management
Using data-aware user experience modeling (Liu and Kavakli 2016a) to compress the data, we improve the efficiency of communication. MOS (mean of score) is used to measure the experience. As shown in Figure 10, for example, the front view of User A has been derived from many sub-patterns with different MOS. Figure 11 shows different MOS values from 5 to 3, and the corresponding necessary bandwidth is given in Table 2. Considering the need of communication, we choose MOS 4 with the Necessary Average Bandwidth (NAB) 16Mbps to hit the optimal balance between the user experience (QoE) and communication efficiency (QoS).

Confidentiality
As shown in Figure 4, the information captured by the camera is processed to protect privacy. A data-aware confidential communication strategy is used in the system. The information of the users has been ciphered, and the information fingerprint has been given by using HASH. The data fingerprint by HASH is used to implement awareness of agent identification. By matching the fingerprint, the data and its source can be detected, as well as the authorization of the components linked to the system.   As shown in Figure 4, the captured data is ciphered with applying a data-aware confidential PKC strategy . Table 3 shows the fingerprint results by applying MD5 for the user information abstract and SHA1 for the outdoor information abstract respectively. Figure 12 shows the corresponding results of an unauthorized receiver when applying forced decoding of the protected data in Figure 9. It is the same that when outdoor sightseeing is received by an unauthorized user, the information will be in a form as white noise.

Context Pattern Analysis
For mixed reality fusion (MRF), the system must recognize the user as a target of interest (TOI) firstly and then map the user into virtual and mixed reality.
To simplify the implementation, pattern based analysis using a mask is designed. The masks for the front vision (Camera B) and side vision (Camera A) are shown in Figure  13, where the white area refers to the range of interest with high belief, the blue area refers to the range of uncertainty, and the black area the range with low belief. Figure 14 shows an example of context analysis of User A with the original information, texture analysis and pattern information by masking. The texture pattern can be further divided into several subsets. The main optimal patterns of   texture context of the User A in Figure 14 are obtained as shown in Figure 15. As shown in Figure 16, the optimal context pattern is found by the context analysis of the texture subset and mask pattern with maximum belief. By using this result of optimal context pattern, the user can be recognized as a target of interest and differentiated from the background (as shown in Figure 17). It can be seen that the results of user TOI preserve desired level of details while filtering the untrustworthy information. The unique properties of the learning operator guarantee the stability of the solution. This makes the proposed algorithm robust to differences in the smoothness features and priori assumptions. Figure 18 shows the process of modeling. After modeling, the user's model uploaded to produce the stereo models of users (as shown in Figure 19).

Data Management for User-Awareness
Users' facial information is stored in the system database for identification (as shown in Figure 20). A dispersed database is used to store the data of different users. Since the data is distributed, the information can be accessed and updated easily. To test the performance of the identification, we use Standard AT&T (ORL) face database (as shown in Figure  21). Scalable learning strategy (Liu and Kavakli 2016e) is used to train the knowledge system for identification. Figure 22 shows the results of identification. It can be seen that the system identifies the user with similarity matching and loads the stereo model corresponding to the user, thus performing the user-oriented awareness of agent identification.

Scenario Fusion
After TOI processing, the data of a user and an outdoor scene are transferred to process the scene fusion (as shown in Figure 4). The targeted scene is selected as a background for the scene fusion. By pattern-context based analysis and optimal context fusion (Liu and Kavakli 2016c; 2016b), the user's image is synthesized with the target scene, and the result is transferred to displayed. Figure 23 to Figure  25 show the results of scene fusion. It can be seen that user is cut out and smoothly embedded into the target scene, with the desired level of details while filtering the untrustworthy information.
Based on the results above, a recognition based mixed reality fusion using a real-virtual-object (RVO) has been further implemented. As shown in Figure 26, the system can recognize the portrait of users respectively (Figure 26 (a)) and merge the virtual stereo model corresponding to each user in the original view as a real-virtual entity, building a scene of MRF object (Figure 26 (b)). Taking user A, Figure 27 shows the different views of the entity. It can be seen that in the MRF scene, reproduced entity shows a stereo portrait of the user. It also shows that the proposed collaborative information system is efficient. This provides a flexible customization method for scene creation and manipulation.

Conclusion
In this paper, we present the design of an agent aware collaborative information system (CIS) for mixed reality applications with the implementations of related modules. We study the concept of agent aware computing and analyze the system performance of the iHCI framework using agent aware computing. Several types of awareness related to the interactive target,user-experience, system performance, confidentiality, and agent identification, have been realized by applying a number of strategies, such as context pattern analysis, scalable learning, data-aware confidential computing, in the implementation.
Using human-aware computing, system-aware computing, and human-system aware computing, the issues in (Liu and Kavakli 2016d), such as mixed reality fusion, QoE-QoS management, confidentiality, and security, are addressed. Applying agent-aware computing based scheme into an implementation of the collaborative information system for mixed reality, we improve user experience and immersion. Agent-aware computing scheme provides a valid approach to tackle the issues, regarding the lack of understanding of users.
The outputs of each module are given to show the feasibility. The results show that the system successfully performs the tasks including MRF, user experience management, confidential communication, data authentication, and user identification, and performs collaborative information processing for mixed reality. The results also shows the feasibility of the solution to the issues related to dispersed systems, QoE-QoS balance, and privacy management in CIS for mixed reality applications.