Efficient GPU Based Sampling for Scene-Space Video Processing

We describe a method to efficiently collect and filter a large set of 2D pixel observations of unstructured 3D

points, with applications to

scene-space aware

video processing. One of the main challenges in scene-space video

processing is to achieve reasonable computation time despite the very large volumes of data, often in the order of

billions of pixels. The bottleneck is determining a suitable set of candidate samples used to compute each output

video pixel color. These samples are observations of the same 3D point, and must be gathered from a large number

of candidate pixels, by volumetric 3D queries in scene-space. Our approach takes advantage of the spatial and

temporal continuity inherent to video to greatly reduce the candidate set of samples by solving 3D volumetric

queries directly on a series of 2D projections, using out-of-core data streaming and an efficient GPU producer-

consumer scheme that maximizes hardware utilization by exploiting memory locality. Our system is capable of

processing over a trillion pixel samples, enabling various scene-space video processing applications on full HD

video output with hundreds of frames and processing times in the order of a few minutes.