Modeling with dynamic textures



Martin Jagersand, Dana Cobzas, Keith Yerex and Neil Birkbeck



References

VR 2003 Tutorial: Recent Methods for Image-Based Modeling and Rendering

Jagersand, M., Birkbeck, N., Cobzas, D. A Three-tier Hierarchical Model for Capturing and Rendering of 3D Geometry and Appearance from 2D Images , International Symposium on 3D Data Processing, Visualization and Transmission (3DPVT) 2008

Cobzas, D. and Jagersand M. Tracking and Predictive Display for a Remote Operated Robot using Uncalibrated Video, IEEE Conference on Robotics and Automation (ICRA) 2005, pp1859-1864 Best vision paper award

Martin Jagersand, Dana Cobzas, Keith Yerex Modulating View-dependent Textures, Eurographics 2004, pp69-72, short presentation

Cobzas, D. and Jagersand M. A comparison of Viewing Geometries for Augmented Realit, Scandinavian Conference on Image Analysis (SCIA) 2003, Also in LNCS, Springer 2749:501-508

Cobzas, D., Yerex, K. and Jagersand, M. Editing Real World Scenes: Augmented Reality with Image-Based Rendering, IEEE Virtual Reality 2003, poster session

Cobzas, D., Yerex, K. and Jagersand, M., Dynamic Textures for Image-Based Rendering of Fine-Scale 3D Structure and Animation of Non-Rigid Motion, International Journal of the Eurographics Association 21(3):493-502, 2002

Cobzas, D. and Jagersand, M. Tracking and Rendering using Dynamic Textures and Geometric Structure from Motion, Proc. of European Conference on Computer Vision (ECCV) 2002, Also in LNCS, Springer 2352:415-432

Yerex, K., Cobzas, D. and Jagersand, M. Image-based Rendering using Hardware Accelerated Dynamic Textures, Proc. of Western Computing Graphics Symposium 2002, pp113-119

Demos


Reanimation of a flower

Reanimation of an arm with a simple 2D geometry

Reanimation of an arm with a simple 2D geometry



Description

Project page - a more detailed description


Overview of modeling with dynamic texture system


The problem of capturing real world scenes from images and then rendering them is of significant interest in both computer vision and graphics research. It has important applications in architectural reconstruction and visualization, virtual museums, landscape scene capturing for robot navigation, and computer games. There are two major approaches to this problem. Traditional graphics techniques use a detailed piecewise planar geometric model , built manually and textured with images. Current image-based methods offer an alternative modeling approach by not explicitly representing scene geometry but instead sampling the scene ray set (plenoptic function) from a collection of calibrated images.


Comparison of analytical and estimated basis for geometric variability. Plane variability: (a) original quadrilateral; (b) warped texture; (c1),(d1) analytical basis ; (c2),(d2) corresponding recovered basis. Parallax variability: (e) reference texture image; (f) depth map; (g1) analytical basis; (g2) recovered basis by PCA.



We have developed a blend of the two methods, where the inaccuracies in the coarse geometry obtained using structure-from-motion is compensated by an image-based correction ( dynamic texture ). Inspired by spatio-temporal basis used in tracking we developed a way of rendering small geometric changes by modulating a stationary image basis called dynamic texture basis. The basis represents a first order approximation of non-linear dependency of texture with respect to the camera view-point. We formulated the variability on the faces of the macro scale geometric model for plane and parallax geometry as well as view dependent light. In practical applications, we estimate this variability from actual intensity statistics in training images using Principal Component Analysis (PCA). This yields a variability basis similar up to a coordinate transform to the analytical variability. At the time of rendering the texture basis is modulated adding a fine scale (up to few pixels) image-based compensation for the coarseness of the geometric model.


Geometric errors in the model are corrected by the dynamic texture. Left: static texture, Right: dynamic texture.



We implemented the system in C++, Open-GL with NVidia specific extensions to hardware accelerate the dynamic texture blending for real time rendering, and Matlab for the user interface. The rendering runs at 50-150 Hz on a consumer PC. We have demonstrated the accurate capture from images and re-rendering of household flowers with complex and fine scale geometry, the motion and non-rigid deformation of a human arm and hand, as well as the non-Lambertian light variation on a human face. All these are very difficult to capture using traditional modeling techniques.


Back to Dana's research page