Introduction
To test the capabilities of the EduExo Pro as an education and research platform, Auxivo, in collaboration with the FGHR (University of Applied Sciences of the Grisons), conducted an innovation project to develop a prototype for an exoskeleton-based virtual reality (VR) gaming experience that explores new ways of using exoskeletons as control and haptic interfaces to the virtual world of video games.
Within the framework of this project, four students, under the supervision of Prof. Christian Bermes, developed and built a fully functional prototype of such a platform over two semesters.
The first project, conducted by a team of three engineering students as a Bachelor project, focused on developing the overall setup, the EduExo Integration, and the general software and game architecture. The second project conducted by one Bachelor student focused on creating a more elaborate game and adding VR-display integration for maximum immersion.
The resulting prototype incorporates two EduExo Pros, integrated with additional peripherals and displays, to create an exoskeleton-based arcade machine. The following video gives you a quick overview of the final prototype:
Hardware Setup
For the setup, the following main components were used:
Two EduExo Pro to track the user's arm movements and provide haptic feedback.
A racing pedal set for additional input with the feet.
Two handheld buttons were used as additional input devices. These were simple push buttons integrated into a 3D-printed handle and connected to one of the open interfaces of the EduExo Pro.
A PC that runs the games.
A screen to display the games.
A frame made out of aluminum profiles that connects everything. This frame also allows for some size adjustment, e.g., for the exoskeletons, chair, or monitor position.
Optionally: A virtual reality display. We used the Meta Quest 3 headset.
To enable users to control an avatar in the game using two EduExo Pro devices, a interface was implemented to connect the exoskeletons with the PC game. This interface facilitated the exchange of control inputs between the exoskeletons, the game, and the integrated sensors.
An adapted EduExo Pro control software was developed to enable the exoskeleton to receive motor control inputs from an external source. In this case, the control software allows the games to provide haptic feedback based on the player's interactions with the virtual world.
To capture the player's arm movements in greater detail, an additional angle sensor (potentiometer) was integrated into the exoskeleton's shoulder joint, allowing measurement of shoulder movements alongside elbow movements.
Additional sensors, which are available with the EduExo Pro but have not yet been used in the prototype, include an IMU motion sensor in the upper arm for capturing detailed upper arm movements and the EduExo Pro's EMG muscle activity sensors. These EMG sensors could be used to directly control game elements by detecting the player's muscle flexing and relaxing, creating the potential for a uniquely immersive gaming experience.
Games
For game development, Unreal Engine 5 was chosen. It is a modern game engine that is more than capable of creating a wide variety of games. As the tutorial in the EduExo handbook uses the Unity 3D game engine, another very versatile game engine, the team wanted to use this project to test the compatibility of the EduExo Pro hardware with the Unreal Engine.
The Unreal Engine editor was used to design the game worlds and levels. Incorporating additional assets from the Unreal Marketplace proved to be an efficient way to create visually appealing environments.
Various game concepts were discussed and explored, focusing on key considerations such as the meaningful integration of exoskeleton arm control mapping into the virtual world. The concepts emphasized intuitive control of a game avatar by tracking the player's arm movements with the exoskeleton and, ideally, providing haptic feedback to the player through the exoskeleton's actuation. The Unreal Engine Blueprint system was used to implement the game logic, providing a viable alternative to writing source code in C++.
First Game: Exoskeleton-based Beer Pong
In the first project, a “Beer Pong” style game was developed in which the player must throw a ball into cups. The movements of a virtual arm were controlled using the exoskeleton. As this was a first prototype, primarily aimed at testing the technical integration of the EduExo Pro with an Unreal Engine-made game, interaction and immersion were quite limited but sufficient to demonstrate feasibility.
Second game: Exoskeleton Sci-fi Exploration
In the second game, the player controls a human avatar wearing an exoskeleton featuring a gripper module on the left arm for handling boxes and a gun integrated into the right arm for shooting targets. The student chose a jungle environment inspired by the Avatar movies as the setting for the game.
Players control the movements of the exoskeleton arms in the game by mirroring their own arm movements in the real-world exoskeleton. A handheld push button on each side allows players to either activate the gripper to pick up objects or fire the gun. To enhance immersion, haptic feedback in the form of motor movements was implemented, simulating the gun's recoil when fired.
Players control movement in the game using racing pedals. Points are earned by hitting targets with the gun or transporting parcels between designated pickup and drop-off locations. The final score also factors in the time taken to complete the level, adding an additional layer of challenge.
VR - Display Integration
To further enhance immersion, a Meta Quest 3 headset was added to display the virtual world. The Quest’s handheld controllers also replaced the custom-made handheld push buttons and racing pedals, streamlining control of both arm movements and overall player movement within the game.
The added VR integration, combined with wearing both arm exoskeletons for interaction with the game environment, created a truly unique and immersive experience.
Summary and Conclusion
This project turned out to be a great experiment, allowing the Auxivo and the FHGR student team to explore new ideas and technologies for developing a unique exoskeleton research platform. Many aspects worked well, providing players with an enjoyable experience. However, certain areas, such as integrating additional sensors, offer opportunities for further exploration.
When designing games for this platform, it's important to consider the absence of leg exoskeletons (for now), which limits the ability to display natural leg movements. Consequently, the most intuitive and immersive experiences are likely to be cockpit-based games in which players control a vehicle equipped with robotic arms. Imagine a robot on wheels cleaning up a planet, steering a submarine for deep sea exploration, collecting pirate treasure, flying a spaceship harvesting asteroids, or similar creative setups.
If you're interested in conducting a project like this as part of your research or as a student initiative, the EduExo Pro exoskeleton is an excellent starting point. It offers a versatile and adaptable hardware platform to kick-start your journey.
Important Disclaimer: This is not a commercially available product. The hardware setup described in this article is not available for purchase, and we cannot provide any technical documentation or manuals beyond the details shared here. This was a purely exploratory project conducted by several students, and we chose to share it in the hope of inspiring others to explore similar ideas.
Additional Information