Floating Textures

Abstract

We present a novel multi-view, projective texture mapping technique. While previous multi-view texturing approaches lead to blurring and ghosting artefacts if 3D geometry and/or camera calibration are imprecise, we propose a texturing algorithm that warps (``floats'') projected textures during run-time to preserve crisp, detailed texture appearance. Our GPU implementation achieves interactive to real-time frame rates. The method is very generally applicable and can be used in combination with many image-based rendering methods or projective texturing applications. By using Floating Textures in conjunction with, e.g., visual hull rendering, light field rendering, or free-viewpoint video, improved rendering results are obtained from fewer input images, less accurately calibrated cameras, and coarser 3D geometry proxies. In a nutshell, the notion of Floating Textures is to correct for local texture misalignments by determining the optical flow between projected textures and warping the textures accordingly in the rendered image domain. Both steps, optical flow estimation and multi-texture warping, can be efficiently implemented on graphics hardware to achieve interactive to real-time performance.

Read full paper

accompanying video

bibliography

source code

Source requires a GeForce8800GTX, current version produces errors on other GPUs, we are working on this. Sorry.

Introduction

To take advantage of the continuing progress in graphics hardware capabilities for realistic rendering, ever more detailed model descriptions are needed. Because creating complex models with conventional modeling and animation tools is a time-consuming and expensive process, direct modeling techniques from real-world examples are an attractive alternative. By scanning, or reconstructing, the 3D geometry of an object or scene and capturing its visual appearance using photos or video, the goal of direct modeling techniques is to achieve photo-realistic 3D rendering results at interactive frame rates.

Unfortunately, acquiring highly accurate 3D geometry and calibrated images turns out to be at least as tedious and time-consuming as model creation using software tools. In response, a number of different image-based rendering (IBR) techniques have been devised that make do with more approximate geometry. In general, however, the price for contending with approximate 3D geometry is that (many) more input images must be available, else rendering quality degrades and artefacts quickly prevail. Another unsolved challenge is that, while image-based rendering can compensate for coarse 3D geometry if sufficiently many input images are available, all techniques still require well calibrated cameras and input images. Thus, time-consuming camera calibration procedures must precede image-based object acquisition every time. But even with the utmost care taken during acquisition, minute camera calibration inaccuracies, tiny 3D scanning holes, and small registration errors can occur and visibly degrade rendering quality of the model.

What is needed is a multi-image texturing algorithm that achieves best-possible results from imprecisely calibrated images and approximate 3D geometry. We propose a GPU-based multi-view texturing algorithm towards this goal. Because the algorithm runs independently on the graphics card, it can be used in conjunction with many image-based modeling and rendering (IBMR) techniques to improve rendering outcome.

As particular contributions, our paper presents:

• a novel texturing algorithm that constitutes a symbiosis between classical linear interpolation and optical flow-based warping refinement;
• a novel weighting and visibility scheme which significantly reduces artefacts at occlusion boundaries;
• a general algorithm that can be applied in conjunction with many IBMR techniques to improve rendering quality;
• an efficient GPU-based implementation of the proposed algorithm which achieves interactive to real-time rendering frame rates;
• a simple extension to our Floating Textures approach for static scenes which reduces the actual rendering part to a simple texture look-up.

TU Braunschweig - Fakultät für Mathematik und Informatik - Computer Graphics - Martin Eisemann - Floating Textures EG08