Computer Graphics
TU Braunschweig

Temporal Consistent Motion Parallax for Omnidirectional Stereo Panorama Video


Temporal Consistent Motion Parallax for Omnidirectional Stereo Panorama Video

We present a new pipeline to enable head-motion parallax in omnidirectional stereo (ODS) panorama video rendering using a neural depth decoder. While recent ODS panorama cameras record short-baseline horizontal stereo parallax to offer the impression of binocular depth, they do not support the necessary translational degrees-of-freedom (DoF) to also provide for head-motion parallax in virtual reality (VR) applications.

To overcome this limitation, we propose a pipeline that enhances the classical ODS panorama format with 6 DoF free-viewpoint rendering by decomposing the scene into a multi-layer mesh representation. Given a spherical stereo panorama video, we use the horizontal disparity to store explicit depth information for both eyes in a simple neural decoder architecture. While this approach produces reasonable results for individual frames, video rendering usually suffers from temporal depth inconsistencies. Thus, we perform successive optimization to improve temporal consistency by fine-tuning our depth decoder for both temporal and spatial smoothness.

Using a consumer-grade ODS camera, we evaluate our approach on a number of real-world scene recordings and demonstrate the versatility and robustness of the proposed pipeline.


Author(s):Moritz Mühlhausen, Moritz Kappel, Marc Kassubeck, Paul Maximilian Bittner, Susana Castillo, Marcus Magnor
Published:to appear
Type:Article in conference proceedings
Book: ACM Symposium on Virtual Reality Software and Technology (VRST) (ACM)
DOI:10.1145/3385956.3418965
Presented at:ACM Symposium on Virtual Reality Software and Technology (VRST) 2020


@inproceedings{muhlhausen2020temporal,
  title = {Temporal Consistent Motion Parallax for Omnidirectional Stereo Panorama Video},
  author = {M{\"u}hlhausen, Moritz and Kappel, Moritz and Kassubeck, Marc and Bittner, Paul Maximilian and Castillo, Susana  and Magnor, Marcus},
  booktitle = { {ACM} Symposium on Virtual Reality Software and Technology ({VRST})},
  organization = {{ACM}},
  doi = {10.1145/3385956.3418965},
  year = {2020}
}

Authors