Immersive Content Generation from Standard 2D Videos

People

• Kiana Calagari
• Mohamed Hefeeda

Abstract

The aim of this project is to create compelling immersive videos suitable for VR (virtual reality) devices using only standard 2D videos. The focus of the work is on field sports such as soccer, hockey, basketball, etc. Currently the only way to create immersive content is by using multiple cameras and 360 camera rigs. This means that in addition to the already existing standard 2D cameras around the field, an expensive infrastructure should be added and managed in order to shoot and generate immersive content. In this project, however, we propose a more favorable alternative in which we can utilize the content of the already existing standard 2D cameras around the field to generate an immersive video.

The aim is to create compelling immersive videos using only standard 2D videos.

Details

Setup

We assume a setup in which we have at least 3 cameras as follows. Note that such camera setup is a practical setup in capturing and broadcasting field sports and the following cameras usually exist.

1. The main camera, located in the middle of the field. This camera is a rotating camera capturing wide views and following the ball around the field. It is usually the main camera used for broadcasting games, and most of the feed that audience view comes from this camera.
2. A camera on the right side of the field which covers the players on the right that might be missing in the main camera. This camera doesn't necessarily have to be rotating.
3. A camera on the left side of the field which covers the players on the left that might be missing in the main camera. This camera doesn't necessarily have to be rotating.

Process

The main steps for generating an immersive video from 2D cameras around the field can be described as follows:

The main steps of our technique.

1. Generating a still panorama using the motion of the main camera: The viewing angle in regular sports videos is usually not wide enough for an immersive experience. In order to improve the sense of presence, a wider viewing angle is needed. As a result,we increase the viewing angle by utilizing the camera rotation, and generating a panorama image which includes the static parts of the scene. This stage can be performed only once, or periodically during a long game to capture any changes in the background. Only the main video feed is used in this stage. It is recommended to use a shot in which the camera rotates over a large angle and with minimum zoom. The camera rotation is then transformed to a wider viewing angle by aligning the frames using image registration techniques, and applying median filtering.

Example of a static panorama generated from a basketball game.

1. Removing the parallax between cameras.
2. Overlaying the video frames of the main camera on the panorama, and locating and overlaying the missing players using the left and right cameras.

Examples of final panoramas generated by our technique for different games: basketball (top), hockey (middle), and volleyball (bottom). The blue arrows indicate the players that have been copied from the left or right feeds.