Reality CG
Abstract
Scope of "Reality CG" is to pioneer a novel approach to modelling, editing and rendering in computer graphics. Instead of manually creating digital models of virtual worlds, Reality CG will explore new ways to achieve visual realism from the kind of approximate models that can be derived from conventional, real-world imagery as input.
Computer Graphics of the Real World - Realistic Rendering, Modelling, and Editing of Dynamic, Complex Natural Scenes
 | ERC Starting Grant No.2569412011 - 2016 |  |
Project Summary
Scope of Reality CG is to pioneer a novel approach to modelling, editing and rendering in computer graphics. Instead of manually creating digital models of virtual worlds, Reality CG will explore new ways to achieve visual realism from the kind of approximate models that can be derived from conventional, real-world imagery as input.
Today's state-of-the-art 3D reconstruction methods from computer vision are able to estimate digital models of a wide variety of real-world, dynamic scenes from multi-view video recordings. Inevitable model reconstruction inaccuracies, however, lead to rendering artefacts, and missing model editing capabilities so far prevent the wide-spread use of real world-based models for realistic computer graphics. Reality CG aims at overcoming these limitations. The goal is to demonstrate that realistic rendering, modelling and editing from real world-acquired dynamic scenes is a viable and advantageous alternative to conventional 3D digital content creation.
The project is motivated by the continuously increasing demand for visual realism in many application areas of computer graphics. Recent advances in graphics hardware and algorithms have made it possible to achieve realistic rendering results in real-time as long as the digital models to be rendered are realistically detailed. Using today's 3D modelling tools, however, the degree of model detail scales roughly linearly with the amount of time invested into manual model design. As a result, the traditional, labour-intensive process of 3D digital content creation threatens to stall further progress in realistic computer graphics applications and new visual media.
Reality CG addresses this precarious modelling bottleneck. To find viable solutions, the project involves and inter-connects three different areas of visual research: Reality CG makes use of the sophisticated mathematical methods developed in computer vision and combines them with knowledge from visual perception to develop new techniques for realistic modelling, editing, and rendering in computer graphics.
Over the course of the project, Reality CG will provide the enabling technology to open up the real world to computer graphics methodology and applications. By extending the scope of computer graphics beyond virtual content, the project will make a profound impact on the field of visual computing, pioneering new research directions as well as breaking ground for novel applications.
Researchers
Affiliated Researchers
Alumni
Publications
Adaptive Image-Space Sampling for Gaze-Contingent Real-time Rendering
Poster @ German Conference on Pattern Recognition 2016, September 2016.
Gaze-contingent Computational Displays: Boosting perceptual fidelity
in IEEE Signal Processing Magazine, vol. 33, no. 5, IEEE, pp. 139-148, September 2016.
Adaptive Image-Space Sampling for Gaze-Contingent Real-time Rendering
in Computer Graphics Forum (Proc. of Eurographics Symposium on Rendering EGSR), vol. 35, no. 4, pp. 129-139, July 2016.
EGSR'16 Best Paper Award
Interactive Scene Flow Editing for Improved Image-based Rendering and Virtual Spacetime Navigation
in Proc. ACM Multimedia, ACM, pp. 631-640, October 2015.
An Affordable Solution for Binocular Eye Tracking and Calibration in Head-mounted Displays
in Proc. ACM Multimedia, pp. 15-24, October 2015.
Won the 'Best Student Paper Award'.
Web-based Interactive Free-Viewpoint Streaming
in Proc. ACM Multimedia, pp. 1031-1034, October 2015.
Poster Presentation
An Affordable Solution for Binocular Eye Tracking and Calibration in Head-mounted Displays
Poster @ ACM Multimedia 2015, October 2015.
Efficient GPU Based Sampling for Scene-Space Video Processing
in Proc. Vision, Modeling and Visualization (VMV), pp. 103-110, October 2015.
A Convex Clustering-based Regularizer for Image Segmentation
in Proc. Vision, Modeling and Visualization (VMV), Eurographics Association, pp. 87-94, October 2015.
An Approach Towards Fast Gradient-based Image Segmentation
in IEEE Transactions on Image Processing (TIP), vol. 24, no. 9, pp. 2633-2645, September 2015.
Sampling Based Scene-Space Video Processing
in ACM Transactions on Graphics (Proc. of Siggraph), vol. 34, no. 4, pp. 67:1-67:11, August 2015.
ACM Siggraph 2015 paper
Interactive Spacetime Reconstruction in Computer Graphics
PhD thesis, TU Braunschweig, July 2015.
ElectroEncephaloGraphics: a Novel Modality for Graphics Research
PhD thesis, TU Braunschweig, July 2015.
Reconstruction of Dense Correspondences
in Marcus A. Magnor, Oliver Grau, Olga Sorkine-Hornung, Christian Theobalt (Eds.): Digital Representations of the Real World: How to Capture, Model, and Render Visual Reality, CRC Press, ISBN 9781482243819, pp. 113-133, May 2015.
Image- and Video-based Rendering
in Marcus A. Magnor, Oliver Grau, Olga Sorkine-Hornung, Christian Theobalt (Eds.): Digital Representations of the Real World: How to Capture, Model, and Render Visual Reality, CRC Press, ISBN 9781482243819, pp. 261-280, May 2015.
Digital Representations of the Real World: How to Capture, Model, and Render Visual Reality
A K Peters/CRC Press, ISBN 9781482243819, May 2015.
Cloth Modeling
in Marcus Magnor and Oliver Grau and Olga Sorkine-Hornung and Christian Theobalt (Eds.): Digital Representations of the Real World: How to Capture, Model, and Render Visual Reality, CRC Press, pp. 229-243, May 2015.
Stereo 3D and Viewing Experience
in Marcus Magnor and Oliver Grau and Olga Sorkine-Hornung and Christian Theobalt (Eds.): Digital Representations of the Real World: How to Capture, Model, and Render Visual Reality, CRC Press, ISBN 9781482243819, pp. 281-295, May 2015.
Temporal Video Filtering and Exposure Control for Perceptual Motion Blur
in IEEE Transactions on Visualization and Computer Graphics (TVCG), vol. 21, no. 5, pp. 663-671, May 2015.
10.1109/TVCG.2014.2377753
A Nonobscuring Eye Tracking Solution for Wide Field-of-View Head-mounted Displays
Technical Demo, March 2015.
IEEE VR, won the 'Honorable Mention' for Technical Demos.
ElectroEncephaloGraphics: Making Waves in Computer Graphics Research
in IEEE Computer Graphics and Applications, vol. 34, no. 6, pp. 46-56, November 2014.
Garment Replacement in Monocular Video Sequences
in ACM Transactions on Graphics, vol. 34, no. 1, pp. 6:1-6:10, November 2014.
Monocular Albedo Reconstruction
in Proc. IEEE International Conference on Image Processing (ICIP), IEEE, pp. 1046-1050, October 2014.
Compressed Manifold Modes for Mesh Processing
in Computer Graphics Forum (Proc. of Symposium on Geometry Processing SGP), vol. 33, no. 5, Eurographics Association, pp. 35-44, July 2014.
Correspondence and Depth-Image Based Rendering: a Hybrid Approach for Free-Viewpoint Video
in IEEE Trans. Circuits and Systems for Video Technology (T-CSVT), vol. 24, no. 6, pp. 942-951, June 2014.
A Nonobscuring Eye Tracking Solution for Wide Field-of-View Head-mounted Displays
Poster @ Eurographics 2014, April 2014.
Sparse Localized Deformation Components
in ACM Transactions on Graphics (Proc. of Siggraph Asia), vol. 32, no. 6, pp. 179:1-179:10, November 2013.
Cost Volume-based Interactive Depth Editing in Stereo Post-processing
in Proc. European Conference on Visual Media Production (CVMP), vol. 10, pp. 1-6, November 2013.
A Survey on Time-of-Flight Stereo Fusion
in M. Grzegorzek and C. Theobalt and R. Koch and A. Kolb (Eds.): Time-of-Flight and Depth Imaging, Springer, pp. 105-127, September 2013.
Optimizing Apparent Display Resolution Enhancement for Arbitrary Videos
in IEEE Transactions on Image Processing (TIP), vol. 22, no. 9, pp. 3604-3613, September 2013.
Patent number 10 2013 105 638.
Virtual Video Camera: a System for Free Viewpoint Video of Arbitrary Dynamic Scenes
PhD thesis, TU Braunschweig, June 2013.
Detail Hallucinated Image Interpolation
Master's thesis, TU Braunschweig, May 2013.
A Framework for Image-Based Stereoscopic View Synthesis from Asynchronous Multi-View Data
in Emerging Technologies for 3D Video: Creation, Coding, Transmission and Rendering, Wiley, ISBN 978-1-118-35511-4, pp. 249-270, May 2013.
Capture and Statistical Modeling of Arm-Muscle Deformations
in Computer Graphics Forum (Proc. of Eurographics EG), vol. 32, no. 2, pp. 285-294, May 2013.
High Resolution Image Correspondences for Video Post-Production
in Journal of Virtual Reality and Broadcasting (JVRB), vol. 9.2012, no. 8, pp. 1-12, December 2012.
Integrating Approximate Depth Data into Dense Image Correspondence Estimation
in Proc. European Conference on Visual Media Production (CVMP), vol. 9, pp. 1-6, December 2012.
Improving Dense Image Correspondence Estimation with Interactive User Guidance
in Proc. ACM Multimedia, ACM, pp. 1129-1132, October 2012.
Pattern- and Markerless RGB and Depth Sensor Calibration
Bachelor thesis, October 2012.
High Detail Marker based 3D Reconstruction by Enforcing Multiview Constraints
Poster @ SIGGRAPH 2012, August 2012.
SIGGRAPH '12: ACM SIGGRAPH 2012 Posters
Single Trial EEG Classification of Artifacts in Videos
in ACM Transactions on Applied Perception, vol. 9, no. 3, pp. 12:1-12:15, July 2012.
A Loop-Consistency Measure for Dense Correspondences in Multi-View Video
in Journal of Image and Vision Computing, vol. 30, no. 9, pp. 641-654, June 2012.
EEG Analysis of Implicit Human Visual Perception
in Proc. ACM Human Factors in Computing Systems (CHI), pp. 513-516, May 2012.
A Toolchain for Capturing and Rendering Stereo and Multi-View Datasets
in Proc. The International Conference on 3D Imaging (IC3D), pp. 1-7, December 2011.
Making of ”Who Cares?” HD Stereoscopic Free Viewpoint Video
in Proc. European Conference on Visual Media Production (CVMP), vol. 8, pp. 1-10, November 2011.
Towards Plenoptic Raumzeit Reconstruction
in Cremers, D. and Magnor, M. and Oswald, M.R. and Zelnik-Manor, L. (Eds.): Video Processing and Computational Video, Springer, ISBN 978-3-642-24869-6, pp. 1-24, October 2011.
Two Algorithms for Motion Estimation from Alternate Exposure Images
in Cremers, D. and Magnor, M. and Oswald, M.R. and Zelnik-Manor, L. (Eds.): Video Processing and Computational Video, Springer, ISBN 978-3-642-24869-6, pp. 25-51, October 2011.
Video Processing and Computational Video
Springer, ISBN 978-3-642-24869-6, October 2011.
Object-aware Gradient-Domain Image Compositing
in Proc. Vision, Modeling and Visualization (VMV), pp. 65-71, October 2011.
Markerless Motion Capture using multiple Color-Depth Sensors
in Proc. Vision, Modeling and Visualization (VMV), pp. 317-324, October 2011.
Monocular Pose Reconstruction for an Augmented Reality Clothing System
in Proc. Vision, Modeling and Visualization (VMV), pp. 339-346, September 2011.
Assessing the Quality of Compressed Images Using EEG
in Proc. IEEE International Conference on Image Processing (ICIP), pp. 3170-3173, September 2011.
Flowlab - an interactive tool for editing dense image correspondences
in Proc. European Conference on Visual Media Production (CVMP), August 2011.
Evaluation of Video Artifact Perception Using Event-Related Potentials
in Proc. ACM Applied Perception in Computer Graphics and Visualization (APGV), p. 5, August 2011.
Integrating multiple depth sensors into the virtual video camera
in Proc. SIGGRAPH, ACM, p. 1, August 2011.
SIGGRAPH '11: ACM SIGGRAPH 2011 Posters
Motion Field Estimation from Alternate Exposure Images
in IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), vol. 33, no. 8, pp. 1577-1589, August 2011.
Dense Correspondence Field Estimation from Multiple Images
PhD thesis, TU Braunschweig, June 2011.
Monsenstein und Vannerdat, ISBN 978-3-86991-339-1
Stereoscopic 3D view synthesis from unsynchronized multi-view video
in Proc. European Signal Processing Conference (EUSIPCO), pp. 1904-1909, May 2011.
Edge-Constrained Image Compositing
in Proc. Graphics Interface (GI), pp. 191-198, May 2011.
The virtual video camera: Simplified 3DTV acquisition and processing
in Proc. 3DTV-CON, IEEE Computer Society, pp. 1-4, May 2011.
Correspondence Estimation and Image Interpolation for Photo-Realistic Rendering
PhD thesis, TU Braunschweig, April 2011.
Robust Feature Matching in General Multi-Image Setups
in Journal of WSCG, vol. 19, pp. 1-8, February 2011.
Perception-motivated interpolation of image sequences
in ACM Transactions on Applied Perception, vol. 8, no. 2, pp. 1-25, February 2011.
Related Projects
Digital Representations of the Real World
The book presents the state-of-the-art of how to create photo-realistic digital models of the real world. It is the result of work by experts from around the world, offering a comprehensive overview of the entire pipeline from acquisition, data processing, and modelling to content editing, photo-realistic rendering, and user interaction.
This project focuses on using electroencephalography (EEG) to analyze the human visual process. Human visual perception is becoming increasingly important in the analyses of rendering methods, animation results, interface design, and visualization techniques. Our work uses EEG data to provide concrete feedback on the perception of rendered videos and images as opposed to user studies that just capture the user's response. Our results so far are very promising. Not only have we been able to detect a reaction to artifacts in the EEG data, but we have also been able to differentiate between artifacts based on the EEG response.
Eye-tracking Head-mounted Display
Immersion is the ultimate goal of head-mounted displays (HMD) for Virtual Reality (VR) in order to produce a convincing user experience. Two important aspects in this context are motion sickness, often due to imprecise calibration, and the integration of a reliable eye tracking. We propose an affordable hard- and software solution for drift-free eye-tracking and user-friendly lens calibration within an HMD. The use of dichroic mirrors leads to a lean design that provides the full field-of-view (FOV) while using commodity cameras for eye tracking.
We present a novel multi-view, projective texture mapping technique. While previous multi-view texturing approaches lead to blurring and ghosting artefacts if 3D geometry and/or camera calibration are imprecise, we propose a texturing algorithm that warps (``floats'') projected textures during run-time to preserve crisp, detailed texture appearance. Our GPU implementation achieves interactive to real-time frame rates. The method is very generally applicable and can be used in combination with many image-based rendering methods or projective texturing applications. By using Floating Textures in conjunction with, e.g., visual hull rendering, light field rendering, or free-viewpoint video, improved rendering results are obtained from fewer input images, less accurately calibrated cameras, and coarser 3D geometry proxies. In a nutshell, the notion of Floating Textures is to correct for local texture misalignments by determining the optical flow between projected textures and warping the textures accordingly in the rendered image domain. Both steps, optical flow estimation and multi-texture warping, can be efficiently implemented on graphics hardware to achieve interactive to real-time performance.
Image-space Editing of 3D Content
The goal of this project is to develop algorithms in image space that allow photo-realistic editing of dynamic 3D scenes. Traditional 2D editing tools cannot be applied to 3D video as in addition to correspondences in time spatial correspondences are needed for consistent editing. In this project we analyze how to make use of the redundancy in multi-stereoscopic videos to compute robust and dense correspondence fields. these space-time correspondences can then be used to propagate changes applied to one frame consistently to all other frames in the video. Beside the transition of classical video editing tools we want to develop new tools specifically for 3D video content.
This project has been funded by ERC Grant #256941 `Reality CG` and the German Science Foundation, DFG MA2555/4-2.
Motivated by the advent of mass-market head-mounted immersive displays, we set out to pioneer the technology needed to experience recordings of the real world with the sense of full immersion as provided by VR goggles.
This project investigates multi-camera setups using Microsoft Kinects. Active structured light from the Kinect is used in several scenarious, including gas flow description, motion capture and free-viewpoint video.
While the ability to capture depth alongside color data (RGB-D) is the starting point of the investigations, the structured light is also used more directly. In order to combine Kinects with passive recording approaches, common calibration with HD cameras is also a topic.
Perception-motivated Interpolation of Image Sequences
We present a method for image interpolation which is able to create high-quality, perceptually convincing transitions between recorded images. By implementing concepts derived from human vision, the problem of a physically correct image interpolation is relaxed to an image interpolation that is perceived as physically correct by human observers. We find that it suffices to focus on exact edge correspondences, homogeneous regions and coherent motion to compute such solutions. In our user study we confirm the visual quality of the proposed image interpolation approach. We show how each aspect of our approach increases the perceived quality of the interpolation results, compare the results obtained by other methods and investigate the achieved quality for different types of scenes.
The high degree of redundancy in video footage allows compensating for noisy depth estimates and to achieve various high-quality processing effects such as denoising, deblurring, super resolution, object removal, computational shutter functions, and scene-space camera effects.
Goal of this project is to assess the quality of rendered videos and especially detect those frames that contain visible artifacts, e.g. ghosting, blurring or popping.
The Virtual Video Camera research project is aimed to provide algorithms for rendering free-viewpoint video from asynchronous camcorder captures. We want to record our multi-video data without the need of specialized hardware or intrusive setup procedures (e.g., waving calibration patterns).
Visual Fidelity Optimization of Displays
The visual experience afforded by digital displays is not identical to our perception of the genuine real world. Display resolution, refresh rate, contrast, brightness, and color gamut neither match the physics of the real world nor the perceptual characteristics of our Human Visual System. With the aid of new algorithms, however, a number of perceptually noticeable degradations on screen can be diminished or even completely avoided.
Official music video "Who Cares" by Symbiz Sound; the first major production using our Virtual Video Camera.
Dubstep, spray cans, brush and paint join forces and unite with the latest digital production techniques. All imagery depicts live action graffiti and performance. Camera motion added in post production using the Virtual Video Camera.