# Mathematical and Computational Fundamentals of Visual Appearance

The study of the computational aspects of reflection, and especially the interaction between reflection and illumination, is of fundamental importance in both computer graphics and vision. In computer graphics, the interaction between the incident illumination and the reflective properties of a surface to give its overall visual appearance is a basic building block in most rendering algorithms. We are interested in the mathematical foundations of reflection and appearance, analyzing the theoretical characteristics of illumination, material properties and cast shadows. This analysis can be used to derive new computational algorithms that are relevant to many areas in graphics and vision. This combination of mathematical analysis and signal-processing and other computational tools underlies all of our work (therefore this page also includes a few publications which are also classified under the practical applications to which they pertain).
 Primary Current Participants

Ravi Ramamoorthi
Dhruv Mahajan
Bo Sun
Peter Belhumeur
Ren Ng (Stanford) Fredo Durand (MIT)

 Reflection as Convolution in Spherical Harmonics

My PhD dissertation on A Signal-Processing Framework for Forward and Inverse Rendering introduced the basic idea of reflection as a spherical convolution of the incident illumination and the BRDF of the surface, with a product formula in spherical harmonics analogous to the standard convolution formula in the Fourier domain. While many authors had qualitatively described reflection as a convolution over the last 20 years, this is the first time a formal analysis and convolution formula have been derived. In terms of signal-processing, the incident lighting can be seen as the signal, and the material properties or BRDF of the surface as the filter. There are many applications to both forward and inverse rendering.
 Analysis of Planar Light Fields From Homogeneous Convex Curved Surfaces Under Distant Illumination Proceedings of Human Vision and Electronic Imaging VI (part of Photonics West, 2001), pages 185--198 This relatively simple to read paper considers the 2D case using only Fourier transforms. Full Paper:     gzipped PS (.6M)    PDF (.2M)    Talk:    PDF (.8M) On the relationship between Radiance and Irradiance: Determining the illumination from images of a convex Lambertian object Journal of the Optical Society of America (JOSA A) Oct 2001, pages 2448-2459 This paper considers the 3D Lambertian case using spherical harmonics and derives an analytic formula for the irradiance in terms of the radiance, including the 9 parameter Lambertian BRDF approximation. Full Paper:     PDF (.4M) Correction: In equation 19, there is a small misprint. The last term should be ((n/2)!)^2, not (n!/2)^2 A Signal-Processing Framework for Inverse Rendering This paper is the most mathematical so far and derives the theory for the general 3D case with arbitrary isotropic BRDFs. It also applies the results to the practical problem of inverse rendering under complex illumination. Full Paper:     gzipped PS (3.7M)    PDF (1M)    Talk:    PPT (1.3M) An Efficient Representation for Irradiance Environment Maps: Siggraph 01, pages 497-500 We use the 9 parameter Lambertian BRDF approximation for analytic rendering of diffuse objects under distant illumination specified by environment maps. Full Paper:     gzipped PS (3.4M)    PDF (1M)    Talk:    PPT (1.8M)    Video Frequency Space Environment Map Rendering: Siggraph 02, pages 517-526 We present a new method for real-time rendering of objects with complex isotropic BRDFs under distant natural illumination, as specified by an environment map. Our approach is based on our signal-processing framework. Full Paper:     gzipped PS (4.2M)    PDF (3.3M) A Signal-Processing Framework for Reflection ACM Transactions on Graphics (volume 23(4), Oct 2004, pages 1004-1042) A new theoretical paper presenting a general unified signal-processing framework for 2D, 3D lambertian, 3D isotropic and 3D anisotropic cases, which includes many new results and significantly generalizes the previous articles. Paper:     PDF
 General Applications of Convolution and Frequency Analysis

The idea of reflection as convolution has more general theoretical and practical relevance. In recent work, we have used convolution to analytically derive principal components for Lambertian surfaces under different illumination conditions, explaining many empirical results on lighting-insensitive recognition for the first time. We have also shown that a convolution result can be derived for cast shadows in natural textures (cast shadows are an element not considered in my PhD thesis at all, which looked only at reflection and illumination). The idea of reflection as convolution has also recently been applied for both subsurface scattering and multiple scattering in volumes. While those methods are not directly related, similar mathematical machinery with Legendre polynomials can be used. Most recently, we have derived a new class of frequency domain identities or invariants , that lead to new BRDF/lighting transfer algorithms for relighting, and a new class of invariants to check image consistency, and detect tampering or image splicing. Finally, we have used frequency-based methods to analyze the accuracy of BRDF factorizations, and to develop a Fourier theory for fast rendering of motion blur.
 Analytic PCA Construction for Theoretical Analysis of Lighting Variability, Including Attached Shadows, in a Single Image of a Convex Lambertian Object PAMI Oct 2002, pp 1322-1333. We explain for the first time some classic empirical results on lighting variability, and take a first step toward analyzing many classic vision problems under complex lighting. Paper:     PDF (.8M) A Fourier Theory for Cast Shadows ECCV 04, pages I 146-162 ; PAMI Feb 05 (pages 288-295) We show that cast shadows can be mathematically analyzed for many simple configurations, resulting in a standard convolution formula that can be derived analytically in 2D and analyzed numerically in 3D. The results help explain many effects of lighting variability in 3D textures and suggest new bases for that purpose. Paper:     ECCV 04 ,     PAMI 05 A Theory of Frequency Domain Invariants: Spherical Harmonic Identities for BRDF/Lighting Transfer and Image Consistency ECCV 06, vol IV, pp 41-55, PAMI 30(2), pages 197-213, Feb 2008. We develop new mathematical results based on the spherical harmonic convolution framework for reflection, deriving novel identities, which are the angular frequency domain analogs to common spatial domain invariants such as reflectance ratios. Paper:     PDF (PAMI 08)     PDF (ECCV 06) An Analysis of the In-Out BRDF Factorization for View-Dependent Relighting EuroGraphics Symposium on Rendering, 2008. We analyze the popular in-out BRDF factorization for view-dependent relighting theoretically. We show that for the Phong BRDFs, the factors or eigenmodes are simply spherical harmonic basis functions, and an exact frequency space analysis is possible. The number of terms needed grows linearly with the Phong exponent or quadratically with the frequency cutoff for reflected or half-angle dependence. The theory allows for setting the number of BRDF terms, and also indicates that high frequency specularity is still out of reach of current algorithms. Paper:     PDF     Video (18M) Frequency Analysis and Sheared Reconstruction for Rendering Motion Blur SIGGRAPH 09. Motion blur is crucial for high-quality rendering, but is also very expensive. Our first contribution is a frequency analysis of motionblurred scenes, including moving objects, specular reflections, and shadows. We show that motion induces a shear in the frequency domain, and that the spectrum of moving scenes is usually contained in a wedge. This allows us to compute adaptive space-time sampling rates, to accelerate rendering. Our second contribution is a novel sheared reconstruction filter that is aligned to the first-order direction of motion and enables even lower sampling rates. Paper:     PDF
 Generalized Clebsch-Gordan Triple Products

My dissertation considered only the interaction of reflection and illumination, without cast shadows. When only two quantities are involved, the mathematical analysis is relatively simpler, since orthonormality can be applied to integrals, and simple results like convolution result. However, when three quantities are involved (lighting, BRDF, visibility), we must often consider an integral of a triple product . In spherical harmonics, a machinery of Clebsch-Gordan expansions has been developed, but with a focus on analytic results for low orders rather than efficient evaluation. To our knowledge, there has been very little work on efficiently evaluating triple products in the mathematical literature. We present the first analysis of triple products in a variety of bases, and a very efficient algorithm for Haar wavelets.
 Triple Product Wavelet Integrals for All-Frequency Relighting Siggraph 04, pp 475-485. We propose a new mathematical and computational analysis of pre-computed light transport. We use factored forms, separately pre-computing the effects of visibility and material properties. Rendering then requires computing triple product integrals at each vertex, involving the lighting, visibility and BRDF. Our main contribution is a general analysis of these triple products likely to have broad applicability in computer graphics and numerical analysis. Paper:     PDF (5M)     Video (17M) Affine Double and Triple Product Wavelet Integrals for Rendering ACM Transactions on Graphics 28(2), Article 14, pages 1-17, Apr 2009. Many problems in computer graphics involve integrations of products of functions. Double- and triple-product integrals are commonly used in applications such as all-frequency relighting or importance sampling, but are limited to distant illumination. In contrast, near-field lighting from planar area lights involves an affine transform of the source radiance at different points in space. Our main contribution is a novel affine double- and triple-product integral theory. Paper:     PDF     Video (AVI 16M)
 First Order or Gradient and Local Light Transport Analysis

Frequency domain or spherical harmonic analysis is by definition global, and often limited to distant illumination or no shadowing. Wavelet methods are fast computationally, but do not permit analytic formulae. First order or gradient analysis is completely local, enables spatial and directional lighting variation, object curvature and bump maps. Analytic formulae enable gradient-based image sampling and interpolation even for soft shadows. A further local analysis of light transport reveals the dimensionality in local patches, and the optimal patch size.
 A First Order Analysis of Lighting, Shading, and Shadows ACM Transactions on Graphics, 26(1), Article 2, pages 1-21, Jan 2007. The shading in a scene depends on a combination of many factors, such as how the lighting varies spatially across a surface, how it varies along different directions, the geometric curvature and reflectance properties of objects, and the locations of soft shadows. In this paper, we conduct a complete first order or gradient analysis of lighting, shading and shadows, showing how each factor separately contributes to scene appearance, and when it is important. PDF A Theory of Locally Low Dimensional Light Transport SIGGRAPH 07. We develop a theory of locally low dimensional light transport, to analytically derive the dimensionality of light transport for a local patch. We analyze the eigenvalues for canonical configurations using Szego's eigenvalue theorem. We show mathematically that for symmetric patches of area A, the number of basis functions for glossy reflections increases linearly with A, while for simple cast shadows, it often increases as sqrt(A). There are practical applications to CPCA and other PRT algorithms. Paper:     PDF     Video (30M)