Abstract (Long paper)

Keywords: Facial Expression, Facial EMG signal, Emotion, computer vision system.

In many affective computing paradigms a user’s internal state is used as an implicit control signal in an interaction. In the work presented here, we are exploring the utilization of two measurement techniques commonly used to assess a user’s affective state as an explicit control signal in a navigation task in a virtual environment. Concretely, we are investigating the feasibility of combining a real-time emotional biometric sensing system and a computer vision system for human emotional characterization and controlling a computer game. A user’s “happiness” and “sadness” levels are assessed by combining information from a camerabased computer vision system and electromyogram (EMG) signals from the facial corrugator muscle. Using a purpose-designed 3D flight simulation game, users control their simulated up-down motions using their facial expressions. To assess if combining visual and EMG data improves facial tracking performance, we conduct a user study where users are navigating through the 3D visual environment using the two control systems, trying to collect as many tokens as possible. We compared two conditions: Computer vision system alone, and computer vision system in combination with the EMG signal. The results show that combining both signals significantly increases the users’ performance and reduces task difficulty. However, this performance increase is associated with a reduced usability due to the need to have EMG sensors on one’s forehead. We hope these results from our study can help in future game designs, aid the development of more immersive virtual environments, and offer for alternative input methods where traditional methods are insufficient or unfesasible.

• Saeedeh Bayatpour is a PhD student in the School of Interactive Arts and Technology at Simon Fraser University in Surrey, BC, Canada. She has degrees in computer software engineering from Islamic Azad University (B.Sc.) and Simon Fraser university (M.Sc). Her research interests are in Human Computer Interaction, 3D display systems
  and computer vision.
• Ulysses Bernardet is a postdoctoral fellow at the School of Interactive Arts and Technology of the Simon Fraser University, Vancouver, Canada. He has a background in psychology, computer science and neurobiology, holds a doctorate in psychology from the University of Zurich, and was a postdoctoral fellow and lecturer at the Universitat Pompeu Fabra in Barcelona, Spain. Ulysses follows an interdisciplinary approach that brings together psychology, neurobiology, robotics, and computer science. He is the main author of the large-scale neural systems simulator iqr, and the core contributor to the conceptualization and realization of several complex real-time interactive systems. At the center of Ulysses’ research activity is the development of models of cognition, emotion, and behavior that are capable of interacting with humans in real-time by means of 3D characters or robots.
• Steve DiPaola is director of the Cognitive Science Program at Simon Fraser University, and leads the iVizLab (ivizlab.sfu.ca), a research lab that strives to make computational systems bend more to the human experience by incorporating biological, cognitive and behavior
  knowledge models. Much of the labs work is creating computation models of very human ideals such as expression, emotion, behavior and creativity. He is most known
  for his AI based computational creativity (darwinsgaze.com) and 3D facial expression systems. He came to SFU from Stanford University and before that NYIT Computer Graphics Lab, an early pioneering lab in high-end graphics techniques. He has held leadership positions at Electronic Arts, and Saatchi Innovation. His computational art has been exhibited at the AIR and Tibor de Nagy galleries in NYC, Tenderpixel Gallery in London
  and Cambridge University’s Kings Art Centre. And exhibited at major museums including the Whitney, MIT Museum, and Smithsonian.
• Alexandra Kitson became a research intern at Simon Fraser University in January 2013 and started her Masters in Fall 2014. She has a BSc from the University of British Columbia in Cognitive Systems, a multidisciplinary program that combines psychology, computer science, philosophy, and linguistics. Her research interests involve using an
  interdisciplinary approach to understand human perception and behaviour. In particular, employing technology as a medium to explore the human psyche, create better human computer interfaces, and provide clinical applications.
• Associate Professor Bernhard Riecke joined Simon Fraser University in 2008 after receiving his PhD from Tübingen University and the Max Planck Institute for Biological Cybernetics and working as a postdoctoral fellow at Vanderbilt University and the Max Planck Institute.
  His research approach combines fundamental scientific research with an applied perspective of improving humancomputer interaction. For example, he uses multidisciplinary research approaches and immersive virtual environments to investigate what constitutes effective, robust, embodied and intuitive human spatial cognition, orientation and behaviour as well as presence and immersion. This fundamental knowledge is used to guide the design of novel, more effective human-computer interfaces and interaction paradigms that enable similarly effective processes in computer-mediated environments such as virtual reality, immersive gaming, and multimedia.

Full text (PDF)  p. 451-457