ZOMBIES!!
A truly untethered Virtual Reality experience.
This project started off as a part of the course, CAP6121 - 3D USER INTERFACES FOR GAMES AND VR at UCF. Designed primarily using Unity 5 integrated with OpenCV for Android and Google Cardboard, it presents the user with eerie environment flooded with zombies. You, as a player should eliminate these zombies using your mystical powers of fire and ice, using your own physical hands.
The idea of this project is to have an interactive VR environment with nothing but your own hands to control it. Current technologies employ joysticks or the computer's keyboard for interaction besides having a wired connection to the HMD. With Google VR, an Android phone and Computer Vision based hand recognition, I aim to enable a wireless, localized VR gaming experience.
Procedure:
The project includes two major assets for its intended functionality,
Google VR SDK: https://developers.google.com/vr/unity/
OpenCV for Unity: https://www.assetstore.unity3d.com/en/#!/content/21088
The idea is to detect the user's hands from the Android phone's back camera and extract the hand regions in real time. The process occurs in three steps:
1. Color Space conversion: RGB to HSV or RGB to YCbCr.
2. Skin Thresholding: Depending on the color space, the minimum and maximum skin values are used at run time to extract the skin depending on the user's current environment.
3. Hand Placement and labeling: Depending on the position of the hand in the captured video frame, the hand region is labelled as a 'right' or a 'left' hand.
The presence or absence of hands in the frame carries out interactive actions in the virtual environment. the specific actions and the consequences in the game are as follows:
1. Right hand: Creates and throws a fireball at the zombie. The zombie is thrown back with an explosion and gets burnt.
2. Left hand: Creates and throws an Iceball at the zombie, which freezes it at the spot surrounded by an icicle trap.
3: Both hands: At times when there are multiple zombies attacking the user, it becomes tedious to take out each zombie, thus the user can place both hands in front of the camera and unleash an energy blast which pushes the zombies away from the player.
Design:
All assets were designed using the primitives available in Unity except for the Zombie model. The model is available for download on the Unity's asset store at https://www.assetstore.unity3d.com/en/#!/content/30232
Game UI:
The game apk has two scenes, one for the configuration of various settings for the video processing and the other, the game itself.
The first scene UI is shown below:
There are 6 sections:
1. Video output window: Here the android camera feed is shown depending on the options chosen by the user.
2: Image Processing and resolution: This contains all the parameters for the each frame being processed.
a. Color Mode: It contains 3 modes: RGB2HSV_FULL, BGR2HSV_FULL, and RGB2YCBCR which converts the Webcamtexture/camera feed to the said colorspaces i.e HSV or YCbCr.
b. Display Tex: This allows the user to show which processing stage is displayed in the video output. The stages are: RGB, Process i.e the HSV or YCbCr, Threshold (depending on the sliders) and Hands(red for right hand and green for left hand)
c: Camera Res: This sets the video capture resolution from Full HD to QCIF(160x120).
d: Hist.Eq.: It contains histogram equalization options, i.e. None, Histogram Equalization and Adaptive Histogram Equalization.
e: Clip and Size: These sliders set the 'clip' and 'size' values for the Adaptive HEQ.
f: Area Thresh: This slider sets the minimum area for the extracted contours to be treated as hand regions.
3: Rendering Quality Options: Allows the setting of graphics quality for the game. The settings are the inbuilt graphics quality settings offered by Unity. The Expensive Processes toggle allows the use of Antialiasing in the game.
4. Load/Save and Start Game: The Load/Save buttons save all the menu options and slider values in the Application.persistent path, so that each scene can load the settings even after restarting the application and the Start button loads the game scene.
5. VR Mode Toggle and IP in game: The 'Show Processing in game' toggle allows the user to see the processed output in game so that it gives an idea of where the user's hands are. The VR mode toggle enables or disables the VR Mode in game.
6: HSV/YCbCr min/max sliders: The left three sliders i.e. HL,SL,VL represent the lower threshold bound and the right three sliders, HH,SH,VH represent the higher threshold bound for the skin detection. The output can be seen by selecting the Threshold option from the Display Tex menu.
Here's how the final output looks like when both the hands are detected. These are the settings for the current environment around the player, since only the skin of the hands gets detected and nothing else in the background. These settings can then be saved and the game can be started.
Game Requirements:
The game apk is designed to run on Android Lollipop 5.0 and higher devices with OpenGLES 3.0 support and multicore rendering.
Screenshots:
Here is a fireball being generated when the right hand is held up.
Here is an ice ball being generated when the left hand is held up.
Note that both the fireball and the iceball remain as long as the hand remains detected in the video. They are fired once the hand is removed from the frame, i.e. when the user makes the throwing motion.
A blast of energy occurs when both hands are help up.
Here's a zombie getting burnt by the fireball.
And here's a zombie trapped and frozen surrounded by ice spikes when hit by the ice ball.
