Computer Graphics
TU Braunschweig

Sampling Based Scene-Space Video Processing

Sampling Based Scene-Space Video Processing

Many compelling video processing effects can be achieved if per-pixel depth information and 3D camera calibrations are known. However, the success of such methods is highly dependent on the accuracy of this “scene-space” information. We present a novel, sampling-based framework for processing video that enables high-quality scene-space video effects in the presence of inevitable errors in depth and camera pose estimation. Instead of trying to improve the explicit 3D scene representation, the key idea of our method is to exploit the high redundancy of approximate scene information that arises due to most scene points being visible multiple times across many frames of video. Based on this observation, we propose a novel pixel gathering and filtering approach. The gathering step is general and collects pixel samples in scene-space, while the filtering step is application-specific and computes a desired output video from the gathered sample sets. Our approach is easily parallelizable and has been implemented on GPU, allowing us to take full advantage of large volumes of video data and facilitating practical runtimes on HD video using a standard desktop computer. Our generic scene-space formulation is able to comprehensively describe a multitude of video processing applications such as denoising, deblurring, super resolution, object removal, computational shutter functions, and other scene-space camera effects. We present results for various casually captured, hand-held, moving, compressed, monocular videos depicting challenging scenes recorded in uncontrolled environments.

Author(s):Felix Klose, Oliver Wang, Jean-Charles Bazin, Marcus Magnor, Alexander Sorkine-Hornung
Published:August 2015
Journal:ACM Transactions on Graphics (Proc. of Siggraph) Vol. 34
Presented at:SIGGRAPH 2021
Note:ACM Siggraph 2015 paper
Project(s): Scene-Space Video Processing  Reality CG 

  title = {Sampling Based Scene-Space Video Processing},
  author = {Klose, Felix and Wang, Oliver and Bazin, Jean-Charles and Magnor, Marcus and Sorkine-Hornung, Alexander},
  journal = {{ACM} Transactions on Graphics (Proc. of Siggraph)},
  doi = {10.1145/2766920},
  volume = {34},
  number = {4},
  note = {{ACM} Siggraph 2015 paper},
  pages = {67:1--67:11},
  month = {Aug},
  year = {2015}