Mixed Reality Advantages

Maverick Kuhn ’20

The Gaspee project we’re designing will be full of fascinating scenes and environments that we would love to share with as many people as possible.  Ideally, everyone would have the opportunity to experience the Gaspee project on a true virtual reality platform. Many details and immersive interactions only shine at their full potential when an individual is actually immersed in the 3D experience.  Unfortunately, access to virtual reality technology like the HTC Vive is limited, and many people may not be able to experience the Gaspee at the optimal level of immersion.  In order to provide a way to preview the experience and get a sense for what it may be like, we have decided to create a couple of mixed reality videos – 2D videos that can be experienced on a desktop, but still contain elements of the full VR production.

So how exactly are we able to give a representation of the VR experience in a 2D video?  Valve, the co-creators of the Vive, along with the help of game developers and video producers, came up with an answer earlier this year.  Using a green screen and some tracking tech, they were able transpose players from the real world into the virtual world, displaying the results onto a live separate monitor of recorded video. With mixed reality, it’s possible for onlookers to see what it’s like to be inside the virtual reality world. This allows for more engaging game trailers and provides a way for onlookers to understand what a player is doing – so those wearing the VR headset and swinging their arms around wildly at nothing will look slightly less dumb.

Mixed reality cuts footage of a real player onto the VR scene they are experiencing

Mixed reality works by taking a camera that has video output of some kind, pointing it at a green screen, and strapping a Vive controller to the top of it. The controller is then able to tell the game where the camera is in relation to the player. The game then renders a second viewpoint that matches the viewpoint of the camera in the real world. All that’s left to do is overlay the green screen footage from the camera onto the game footage in a post-production software like Premiere Pro – the image of the real player in virtual reality will appear on a backdrop of the virtual reality scene from the camera’s perspective.

The ability to track the camera means it can move around in 3d space and still capture the right angle within the game. However, in order to get very dynamic movements (i.e. circling around the player), one needs a green screen room, usually three walls and a floor of chroma key. We are going to test the Cave as a possible method to chroma key. It provides multiple walls of the same color (white). It is possible to use the color white to key out an individual from the background although blue or green screen is the best. The point of these colors is to be as different as possible from the individual being filmed, so then we can cleanly remove the background and replace it with the virtual reality footage. The main worry for using a white background as a chroma key is losing the sclera (or white part) of the eye. However, since the user will be wearing a VR headset, losing the sclera won’t be an issue, and so using a white chroma key may be a viable option.

In order to better understand what mixed reality looks like in practice, check out this trailer for Tilt Brush, Google’s hit VR experience that allows you draw in 3D space; https://www.youtube.com/watch?v=TckqNdrdbgk

Pretty cool huh? Notice how the camera does some dolly shots and pans and yet the environment and individual move together as if the player is really in the virtual environment. The mixed reality technique provides a dynamic and engaging visual. Utilizing a similar approach, we hope to produce short videos in order to present the Gaspee experience without requiring an individual to actually enter VR space.  The experience can be conveyed using mixed reality, allowing anyone to watch another player navigate through the virtual world. This flexibility will allow us to advertise our production and deliver the educational experience to more users than possible before.