Logo CG
Virtual Video Camera
Logo TU

Virtual Video Camera

Image-Based Free Viewpoint Video


"Who Cares", a stereoscopic free-viewpoint music video by Symbiz Sound.



Project Summary

The Virtual Video Camera research project is aimed to provide algorithms for rendering free-viewpoint video from asynchronous camcorder captures. We want to record our multi-video data without the need of specialized hardware or intrusive setup procedures (e.g., waving calibration patterns).

While controlling the location and time of the viewpoint, the user should not be able to distinguish synthetically rendered images from the ones originally recorded. Our key idea is to employ an image interpolation scheme that is based on dense pixel correspondences. This separates us from attempts that rely on depth/geometry reconstruction. We are convinced that a strict enforcement of any geometric model will ultimately fail, since failure cases can easily be constructed for these approaches.

Our goal is to provide and constantly improve a complete end-to-end system that includes algorithms on image correspondence estimation, (real-time) rendering, special-effects creation, camera calibration and quality assessment.


Christian Lipski, Felix Klose, Kai Ruhl, and Marcus Magnor:
"Making of ”Who Cares?” HD Stereoscopic Free Viewpoint Video",
in Proc. European Conference on Visual Media Production (CVMP) 2011, vol. 8, pp. 1–10, November 2011.
Part of projects "Virtual Video Camera" and "Who Cares?".
[pdf] [bib] [making of]

We present a detailed blueprint of our stereoscopic free-viewpoint video system. Using unsynchronized footage as input, we can render virtual camera paths in the post-production stage. The movement of the virtual camera also extends to the temporal domain, so that slow-motion and freeze-and-rotate shots are possible. As a proof-of-concept, a full length stereoscopic HD music video has been produced using our approach.

Christian Lipski, Christian Linz, Kai Berger, Anita Sellent, and Marcus Magnor:
"Virtual Video Camera: Image-Based Viewpoint Navigation Through Space and Time",
Computer Graphics Forum, vol. 29, no. 8, pp. 2555–2568, December 2010.
Part of project "Virtual Video Camera".
[pdf] [bib] [video] [youtube]

We present an image-based rendering system to viewpoint-navigate through space and time of complex real-world, dynamic scenes. Our approach accepts unsynchronized, uncalibrated multi-video footage as input. Inexpensive, consumer-grade camcorders suffice to acquire arbitrary scenes, e.g., in the outdoors, without elaborate recording setup procedures, allowing also for hand-held recordings. Acquisition simplification, integration of moving cameras, generalization to difficult scenes, and space-time symmetric interpolation amount to a widely applicable Virtual Video Camera system.

Christian Linz, Christian Lipski, Lorenz Rogge, Christian Theobalt, and Marcus Magnor:
"Space-Time Visual Effects as a Post-Production Process",
in ACM Multimedia 2010 Workshop - 1st International Workshop on 3D Video Processing (3DVP) 2010, vol. 1, pp. 1–6, October 2010.
Part of project "Virtual Video Camera".
[pdf] [bib] [video] [youtube]

Space-time visual effects play an increasingly prominent role in recent motion picture productions as well as TV commercials. Currently, these effects must be meticulously planned before extensive, specialized camera equipment can be precisely positioned and aligned at the set; once recorded, the effect cannot be altered or edited anymore. In this paper, we present an alternative approach to space-time visual effects creation that allows flexible generation and interactive editing of a multitude of different effects during the post-production stage. The approach requires neither expensive, special recording equipment nor elaborate on-set alignment or calibration procedures. Rather, a handful of off-the-shelf camcorders, positioned around a real-world scene, suffice to record the input data. We synthesize various space-time visual effects from unsynchronized, sparse multi-view video footage by making use of recent advances in image interpolation. Based on a representation in a distinct navigation space, our space-time visual effects (STF/X) editor allows us to interactively create and edit on-the-fly various effects such as slow motion, stop motion, freeze-rotate, motion blur, multi-exposure, flash trail, and motion distortion. As the input to our approach consists solely of video frames, various image-based artistic stylizations, such as speed lines and particle effects are also integrated into the editor. Finally, different effects can be combined, enabling the creation of new visual effects that are impossible to record with the conventional on-set approach.


Image Correspondence Estimation

Christian Linz, Christian Lipski, and Marcus Magnor:
"Multi-Image Interpolation based on Graph-Cuts and Symmetric Optical Flow",
in Proc. Vision, Modeling and Visualization (VMV), Siegen, Germany, pp. 115–122, Eurographics Association, November 2010.
Part of project "Virtual Video Camera".
[pdf] [bib]
Christian Lipski, Christian Linz, Thomas Neumann, Markus Wacker, and Marcus Magnor:
"High Resolution Image Correspondences for Video Post-Production",
in Proc. European Conference on Visual Media Production (CVMP), Los Alamitos, CA, USA, vol. 7, pp. 33–39, IEEE Computer Society, November 2010.
http://doi.ieeecomputersociety.org/10.1109/CVMP.2010.12
Part of project "Virtual Video Camera".
[pdf] [bib]
Felix Klose, Christian Lipski, and Marcus Magnor:
"Reconstructing Shape and Motion from Asynchronous Cameras",
in Proc. Vision, Modeling and Visualization (VMV), Siegen, Germany, pp. 171–177, November 2010.
Part of project "Virtual Video Camera".
[pdf] [bib]
Anita Sellent, Christian Linz, and Marcus Magnor:
"Consistent Optical Flow for Stereo Video",
in Proc. IEEE International Conference on Image Processing (ICIP), pp. 1–4, September 2010.
Part of projects "Multi-Image Correspondences" and "Virtual Video Camera".
[pdf] [bib]
Timo Stich, Christian Linz, Georgia Albuquerque, and Marcus Magnor:
"View and Time Interpolation in Image Space",
Computer Graphics Forum (Proc. of Pacific Graphics), vol. 27, no. 7, pp. 1781–1787, February 2008.
Part of projects "Perception-motivated Interpolation of Image Sequences" and "Virtual Video Camera".
[pdf] [bib] [youtube]

Rendering and Compression

Felix Klose, Kai Ruhl, Christian Lipski, Christian Linz, and Marcus Magnor:
"Stereoscopic 3D view synthesis from unsynchronized multi-view video",
in Proc. European Signal Processing Conference, Barcelona, Spain, pp. 1904–1909, May 2011.
Part of projects "Virtual Video Camera" and "Who Cares?".
[pdf] [bib]
Christian Linz, Christian Lipski, and Marcus Magnor:
"Multi-Image Interpolation based on Graph-Cuts and Symmetric Optical Flow",
in Proc. Vision, Modeling and Visualization (VMV), Siegen, Germany, pp. 115–122, Eurographics Association, November 2010.
Part of project "Virtual Video Camera".
[pdf] [bib]
Martin Eisemann, Timo Stich, and Marcus Magnor:
"3-D Cinematography with approximate and no geometry",
in Rémi Ronfard and Gabriel Taubin (Eds.): Image and Geometry Processing for 3-D Cinematography, Springer, ISBN 978-3-642-12391-7, pp. 259–284, July 2010.
Part of project "Virtual Video Camera".
[pdf] [bib]
Benjamin Meyer, Christian Lipski, Björn Scholz, and Marcus Magnor:
"Multi-view Coding with Dense Correspondence Fields",
in Proc. IEEE International Symposium on Consumer Electronics (ISCE), pp. 117–120, June 2010.
Part of project "Virtual Video Camera".
[pdf] [bib] [windows demo]
Benjamin Meyer, Christian Lipski, Björn Scholz, and Marcus Magnor:
"Real-time Free-Viewpoint Navigation from Compressed Multi-Video Recordings",
in Proc. 3D Data Processing, Visualization and Transmission (3DPVT), May 2010.
Part of project "Virtual Video Camera".
[pdf] [bib]

Calibration

Benjamin Meyer, Timo Stich, Marcus Magnor, and Marc Pollefeys:
"Subframe Temporal Alignment of Non-Stationary Cameras",
in Proc. British Machine Vision Conference (BMVC), September 2008.
Part of project "Virtual Video Camera".
[pdf] [bib]

Quality Assessment

Kai Berger, Christian Lipski, Christian Linz, Anita Sellent, and Marcus Magnor:
"A ghosting artifact detector for interpolated image quality assessment",
in Proc. IEEE International Symposium on Consumer Electronics (ISCE), June 2010.
Part of projects "Virtual Video Camera" and "Virtual Video Quality".
[pdf] [bib]
Kai Berger, Christian Lipski, Christian Linz, Anita Sellent, and Marcus Magnor:
"A ghosting artifact detector for interpolated image quality assessment",
in Proc. ACM Applied Perception in Computer Graphics and Visualization (APGV), September 2009.
Part of projects "Virtual Video Camera" and "Virtual Video Quality".
[pdf] [bib]

Technical Reports, Posters and Student Theses

Felix Klose, Christian Linz, Christian Lipski, and Marcus Magnor:
"Flexible Stereoscopic 3D Content Creation of Real World Scenes",
Technical Report, Computer Graphics Lab, TU Braunschweig, November 2010.
Part of project "Virtual Video Camera".
[bib]
Christian Lipski, Christian Linz, and Marcus Magnor:
"Belief propagation optical flow for high-resolution image morphing",
August 2010.
SIGGRAPH '10: ACM SIGGRAPH 2010 Posters
Part of project "Virtual Video Camera".
[pdf] [bib]
Lorenz Rogge:
"Integration of visual effects into the Virtual Video Camera system",
Master's thesis, Institut für Computergraphik, TU Braunschweig, December 2009.
Part of project "Virtual Video Camera".
[pdf] [bib]
Christian Lipski, Georgia Albuquerque, Timo Stich, and Marcus Magnor:
"Spacetime Tetrahedra: Image-Based Viewpoint Navigation through Space and Time",
Technical Report no. 12-9, Institut für Computergraphik, TU Braunschweig, December 2008.
http://www.digibib.tu-bs.de/?docid=00023968
Part of project "Virtual Video Camera".
[pdf] [bib]

Related Projects (other workgroups)


Line
TU Braunschweig - Fakultät für Mathematik und Informatik - Computer Graphics - Research Projects - Virtual Video Camera