Advanced Material Rendering in Blender

(cid:178) Physically correct and realistic visual appearance rendering or analysis of material surface visual properties require complex descriptive models capable of modelling material dependence on variable illumination and viewing conditions. The recent most advanced representation of visual properties of surface materials is a Bidirectional texture function (BTF). BTF is 7D function of planar coordinates, spectral coordinate, and viewing and illumination angles, respectively. Unlike smooth textures, it specifies their altering appearance due to varying illumination and viewing conditions. This BTF visual appearance dependency on viewing and illumination conditions significantly complicates not only its acquisition, representation, and modelling but also makes its rendering noticeably more demanding. BTF textures are acquired by costly measurements of real materials and their subsequent nontrivial processing. While several techniques for measurement or processing of BTF textures have been described already, there is no environment allowing BTF texture rendering. This contribution describes novel Blender texture plugin for the purpose of BTF texture mapping and rendering. The plugin benefits from our previously developed BTF Roller texture enlargement method which is the integral part of its implementation. The presented plugin allows creating realistic computer animations with additional BTF textures of any required size mapped onto an object surfaces while the other functionality of Blender retains.


INTRODUCTION
There is neither professional nor any open source 3D graphics application currently available which enables realistic surface materials rendering in its most advanced BTF representation form [3], [5]. However an ever-growing number of real world computer vision applications require realistic rendering of genuine materials which cannot be achieved without this recently most advanced available surface material representation. A simple alternative would be to write a proprietary BTF shader, however to develop essential 3D graphics environment for BTF rendering would be very complicated and resources demanding and probably even all options offered by contemporary 3D graphics applications could not be achieved. Thus an existing graphical application suitable for BTF texture rendering enhancement is the appropriate solution. As it turned out the best choice is the 3D graphics application Blender (http://www.blender.org, see Fig.  1). The Blender is open source software which is being actively developed under the supervision of the Blender Foundation and the source codes written in C++ are freely available for further elaboration. Fig. 1. A 3D drapery model created and textured using UV-mapping in Blender can be easily coated with a BTF material measurements thanks to our BTF texture plugin.
The rest of the paper is organized as follows. Section 2 describes the reflectance function visualization problems, while section 3 proposes a modification of the Blender's renderer to be able to calculate viewing and illumination angles for texturing purposes, and extension of Blender's texture plugin interface. Section 4 provides an insight into the BTF texture plugin architecture. Section 5 details the BTF roller underlying algorithm. Section 6 illustrates some visualization results achieved with the implemented rendered and compares them with the classic inadequate approach where the smooth planar textures were used instead. Finally, the concluding section 7 sums up our achievement and suggest some future development of the renderer.

II. REFLECTANCE REPRESENTATION AND VISUALIZATION
Real surface material reflectance is very complex phenomenon which depends on too many (sixteen) physical variables to be able to measure or model all of them and all their possible interactions accurately using available tools. Even the recent most advanced approximation of the general reflectance function the bidirectional texture function (BTF) requires the state-of-the-art [5] nontrivial mathematical models, measurement systems and computer technology both for

Martin Hatka and Michal Haindl
Institute of Information Theory and Automation of the ASCR, Prague, Czech Republic 1 16 The International Journal of Virtual Reality, 2012, 11 (2): [15][16][17][18][19][20][21][22][23] processing as well as data storage and visualization. Multispectral BTF neglects light transport time, variable radiance along light rays, and light transmittance among others. BTF is then a seven-dimensional function, which considers measurement dependency on colour spectrum, planar material position, as well as its dependence on illumination and viewing angles: where the multiindex r = [r 1 , r 2 , r 3 ] specifies planar horizontal and vertical position in material sample image, r 3 is the spectral index and , are elevation and azimuthal angles of the illumination and view direction vector (see Fig. 2). The BTF measurements comprise a whole hemisphere of light and camera positions in the observed material sample coordinates according to the selected quantization steps (see Fig. 3). A fast BTF synthesis method with substantial compression is essential for applications requiring accurate realtime rendering of these data using graphics hardware. In addition, the original BTF measurements only cannot be used in any practical application due to missing necessary measurements from all arbitrary vantage points under arbitrary illumination and due to their small size. Thus, a seamless spatial enlargement (modelling) method of this otherwise huge BTF data is inevitable and also constitutes an integral part of our BTF plugin.

BTF Visualization
Applying BTF textures to 3D models surfaces dramatically enhances the visual appearance of the objects in a rendered scene. Such texturing is the best and physically correct way to achieve photo-realistic results. Accurate texture mapping is essential to get a high quality visualization. Suitable for BTF texturing is the UV-mapping technique which projects a texture map onto a 3D object while the texture map is handled manually. If the accurate UV-mapping of the texture is done, the BTF application to the surface is straightforward. UV texture coordinates unambiguously define the position, the orientation and the scale of the texture on the surface of an object.

Blender
Blender is the free open source 3D graphics application for creation 3D models, visualizations and animations. Blender is available for all major operating system under the GNU General Public License and it is being actively developed. The purpose of Blender is to model and render 3D computer graphics and animations using various techniques such as raytracing, radiosity, ambient occlusion or scanline rendering. Modelling techniques are primarily aimed at facet representation of the objects. Bezier curves or NURBS surfaces are supported as well. Animation capabilities incorporate keyframed animation tools including inverse kinematics, armature, curve and lattice-based deformations, fluid dynamics, and a particle system with collision detection.
Blender, as is, does not handle the dependency of the texture appearance on the lighting conditions. On the other hand, Blender provides an interface for texture plugins. Texture plugin is a dynamically loaded library that exists as a separate file on a computer. When called in it communicates with Blender through given interface to generate the texture.
Overall scheme of the proposed solution to BTF texture rendering in Blender is summarized in the Fig. 4. Fig. 4. BTF rendering using Blender, texture plugin, and the BTF Roller texture synthesis algorithm. Texture analysis using the BTF Roller (yellow box) can be done independently before the rendering. Texture synthesis implemented in the texture plugin is performed as inseparable part of the rendering process (red box). BTF tiles are generated and stored and they are subsequently reused during the rendering.

III. BLENDER MODIFICATION
In the Blender's rendering pipeline there is no dependency of the texture appearance on the viewing and illumination conditions considered. Although several types of diffuse and specular shaders are implemented and these shaders use the surface normal, viewing direction and illumination direction, the texturing is performed earlier than the shading. The utilization of the shaders for the purpose of the BTF texturing has not any sense. This is the reason why the BTF texturing should be solved in different way. The way is to involve the capability to vary the texture appearance on the illumination and viewing conditions directly in the texturing process.
Because of the huge amount of BTF data, BTF support directly in Blender seems to be a very complex, difficult and ineffective task. On the other hand, the texture plugin seems to be an interesting and much simpler way to incorporate BTF textures to Blender. The texture plugin communicates with Blender through the standardized interface and texture data can be treated outside Blender. The plugin interface has to be able to handle the dependency of the texture on the viewing and illumination conditions. The first task is to extend the Blender's renderer to compute the viewing and illumination azimuthal and elevation angles. The second task is to modify the plugin interface to be able to pass these angles to the texture plugin.

Rendering Pipeline Viewing and Illumination Angle Computation
The only necessary modification of the renderer was to incorporate an evaluation of elevation and azimuthal angles of the illumination and view direction. The principle of the modification is as follows.
Each rendered pixel belongs to a triangular facet. In view of the fact that the spatial coordinates and UV texture coordinates of the triangle vertices are known, the UV texture coordinates of the rendered pixel can be evaluated. Moreover, view and illumination vectors are known. Projection of the view and illumination vector on the axes of the texture space provides the coordinates of the view and illumination vector in the texture space. Finally, the elevation and azimuthal angles of the view and illumination vector are calculated.
Let's consider any rendered 3D object which consists of an arbitrary number of triangular facets. The scene is rendered in pixel by pixel order and each rendered pixel belongs to a certain facet of the rendered object. Further, let's consider the rendered pixel V which belongs to the facet V 1 V 2 V 3 (see Fig. 5). The spatial coordinates of the vertices is the orthonormal basis of the 3D texture coordinates. Note that the triangular facet V 1 V 2 V 3 is coplanar with the plane given by the texture axes u and v and its normal n is parallel with the axis w of the texture coordinates space. Because the relationship between the spatial coordinates and the texture coordinates is known (the UV-mapping is done in a 3D model), the axes of the texture space can be expressed in the spatial coordinates, i.
As the next step, let's denote the view and illumination directions expressed in spatial coordinates as d v and d l respectively. Then the projection of a vector d in the directions of the axes t x , t y and t z of the texture coordinates expressed in the spatial coordinates yields in a vector s . Vector s corresponds to the vector d expressed in the texture coordinates, i.e. s = (v u , v v , v w ) = (d ) T . Finally, while the vector s = (d ) T expressed in the texture coordinates is known, required azimuthal angle and elevation angle can be easily calculated from the relationship between the cartesian and spherical coordinates:

BTF Texture Plugin Interface
The texture plugin takes 3D texture coordinate vector (u, v, w) as the input and returns back the vector (y, Y R , Y G , Y B , N u , N v , N w ) as the output, where y is the pixel intensity (in the case of monochromatic texture), Y R , Y G , Y B are the RGB components and N u , N v , N w are the normal vector components in texture coordinates.
As described in section 2, BTF is the 7D function of planar coordinates, spectral coordinate, and viewing and illumination 18 The International Journal of Virtual Reality, 2012, 11 (2): [15][16][17][18][19][20][21][22][23] angles, BTF(r, i , i , v , v ). The input interface was extended to pass the viewing and illumination angles hence the input vector (u, v, w) was substituted by (u, v, w, i where w is not used (only planar textures are considered). To set the texture coordinates u and v, the UV-mapping technique, which is considered as the most accurate, was used.

IV. BTF TEXTURE PLUGIN
BTF texture measurement consists of thousands of colour images (section 2) and to incorporate them directly in Blender is difficult. For that reason it proved to be very effective to take advantage of texture plugin. The proposed BTF texture plugin communicates with Blender through the Blender texture plugin interface Figs.9-11 additionally extended of viewing and illumination azimuthal and elevation angles.
The main advantage of this approach is a possibility to implement various texture synthesis algorithms directly in the plugin, particularly the BTF Roller algorithm (sections 4.3 and 5). The plugin performance optimization is then straightforward.

Barycentric Coordinates Interpolation
BTF textures are measured for a finite number of camera and light source positions, however in practice it is necessary to evaluate pixel values of unmeasured combinations of camera and light source positions. For such a combination interpolation, which is motivated by spherical barycentric coordinates [17], on a triangle with known vertex pixel values is performed. The interpolation using spherical barycentric coordinates would be the most accurate but computationally complex. The following approximation seems to be sufficient.
Let's consider the hemisphere (see Fig. 6) with its center O and the point P on the hemisphere corresponding to the required azimuthal and elevation angle of the view or illumination direction. Let's Y P denotes the value of the desired pixel viewed or illuminated under the direction corresponding to the point P on the hemisphere. Further, denote the three known measured directions, which are closest to the P, as P 1 , P 2 and P 3 and the values of corresponding pixel as Y P1 , Y P2 and Y P3 . Then the value of the pixel Y P will be where w 1 , w 2 and w 3 are the weights of Y P1 , Y P2 and Y P3 , w 1 + w 2 + w 3 = 1.
The weights are defined in the following way: , , , where

BTF Data Buffer
To store the whole BTF texture dataset in memory is ineffective. The best solution is to implement a buffer to store BTF texture data. The buffer size is the user specified parameter. An appropriate software design of the buffer is essential to the plugin performance and the right optimization can dramatically speed up the BTF rendering process.
The buffer is designed as a double linked list (see Fig. 7). This list holds BTF texture image data, each node for the BTF slice corresponding to a particular texture measurement. Moreover, the list is extended by an array of pointers to the list nodes. This optimization allows efficient check if the BTF slice is loaded in the buffer and also fast search in the list. The BTF slices are kept ordered by the last access time and if the memory dedicated to the buffer is exhausted, the first accessed slice is removed from the list.

Texture Synthesis
BTF texture samples have only limited size, typically several hundreds or thousands of pixels. This limited size of texture measurements is the fundamental problem while an object of a 3D scene should be covered with the BTF texture. The simplest way to overcome this problem is to tile the texture sample. On the other hand, the tiled texture has remarkable seams which dramatically decrease the resulting texture visual quality.
In order to generate top quality BTF texture without visible seams in the synthesis step, the texture enlargement sampling type of algorithm, which is called the BTF Roller [7], was incorporated in the plugin (see Fig. 8). The roller method was chosen from possible alternatives ( [4], [19], [15] and many others) because it is the fully automatic and very fast method which produces high quality spatial data enlargement results.
The roller method is based on the overlapping tiling and subsequent minimum error boundary cut. One or several optimal double toroidal data patches are seamlessly repeated during the synthesis step. This efficient method starts with the minimal tile size detection which is limited by the size of control field, the number of toroidal tiles we are looking for, and the sample spatial frequency content. The roller algorithm is described in detail in the following section 5. Then the plugin input is the set of several rectangular and mutually interchangeable BTF texture tiles. These tiles are precomputed only once during the analytical step of the BTF Roller algorithm. Because the resulting texture generated by the plugin has an arbitrary size and is randomly accessed by Blender, it is inefficient to generate whole BTF slices as random tiling. More efficient approach is to generate aperiodic tiling [2]. In our BTF texture plugin we have used the iterative version [18] of the Wang Tiles. Then it could be simply decided which tile and its pixel will be used while the value of the particular pixel from the resulting texture is required by Blender. Alternatively the synthetic BTF textures can be generated from mathematical models [12], 0, [1], [10], 0, [11], [6] which are more flexible and extremely compressed, because only several parameters have to be stored. However, mathematical models can only approximate real BTF measurements, which results in visual quality compromise for some oversimplified methods.
The main advantage of the solution based on the texture plugin is the possibility to implement various BTF texture synthesis methods or various BTF texture models and verify them in commonly used 3D graphics application. Fig. 9. Screenshot of Blender window with 3D scene containing chair model with UV texture coordinates defined (shown using a test textural pattern).

Plugin Parameters
Blender texture plugin interface provides a capability to define the plugin control panel. BTF texture plugin control panel (see Fig. 11) consists of several control elements. The user must specify a path to the BTF texture data, the number of BTF texture tiles used by the synthesis algorithm, and their size. Also the size of the BTF Data Buffer can be set by the user. Finally the size of the resulting texture has to be set by the user. The user can disable data loading for preview purposes and thus speed up the rendering process. Then the BTF texture is not applied to the object. BTF ROLLER ALGORITHM We assume mutually well registered BTF data of the size N by M for changing viewing angle and changing illumination angle. Then the algorithm produces simultaneously identical tiles for all viewing and illumination angles.

Minimal tile size
The minimal rectangle to which the tile is inscribed is limited by the size of BTF measurements, the number n of toroidal tiles we are looking for, and the sample spatial frequency content. From the Fourier transformation of a single monospectral perpendicularly illuminated texture component we detect the dominant low frequency f r we want to preserve. The multiindex r has two components r = [r 1 , r 2 ], the first component is row and the second one is the column index, respectively. The rectangle vertical size is chosen to be , 5 . 0 , 5 . 0 (9) and if we require n>1 number of multiple tiles we add additional condition n row n col NM/n. The horizontal tile size is found similarly.

Overlapping and optimal cut
The double toroidal tile (see Fig. 12) is limited by the selected minimal rectangle to be inscribed in from the original texture measurement. The texture tile is assumed to be indexed on the regular two-dimensional toroidal lattice. The optimal lattice searched by the algorithm allows for seamless repetition in both horizontal and vertical directions, respectively.
Let us define the overlap error for a pixel r as follows: where Y r denotes a multispectral pixel The optimal cuts for both the horizontal and vertical edge are searched in the corresponding overlaps using the dynamic programming method. Alternatively some other suboptimal search such as the A * algorithm can be used if necessary to speed up also the analytical part of the method. The combination of both optimal vertical and horizontal cuts creates the toroidal tile as is demonstrated on Fig. 12.
For more efficient storage and manipulation the resulting tile is converted to the rectangular shape.

Multiple tiles
Some textures with dominant irregular structures cannot be modelled by simple singular tile repetition without clearly visible and visually disturbing regularly repeated effects. These textures are modelled by random ordering of several tiles which have identical tile borders but different internal content. Using analogous approach to the optimal tile border selection we find optimal cut between the optimal tile and new tile filling from another part of the input texture. All multiple tiles have the same size.

Enlargement
The synthesis of any required BTF texture size for a single tile case is simple repetition of the created double toroidal tile in both directions until the required textured area is generated. There is no computation involved in this step hence it can be easily implemented in real time. In the case of several mutually interchangeable tiles uniform random generator is needed to decide which tile will follow. However, this additional computation has negligible overhead.
Within the BTF texture plugin for Blender, the random generator was replaced by an aperiodic tiling generator.

VI. RESULTS
The texture plugin together with the Blender's renderer core modification allow using BTF textures directly in Blender. Especially, UV-mapping of BTF textures and their subsequent rendering can lead to very realistic appearance of any 3D models. We have tested the plugin with BTF measurements either from the University of Bonn [16] or from the Yale University [14].
Figs. 13, 14 demonstrate the application of the BTF texture plugin involving BTF texture synthesis during the visualization of the Ferrari car model interior. Fig. 13. Ferrari 360 Spider car model rendered using BTF textures which allows realistic visualization of 3D scene (3D model courtesy of DMI cars 3D models, http://www.dmi-3d.net/). The advantage of BTF rendering is also demonstrated in Fig.  15. Images in the top row were rendered using BTF texture measurements while the images in the bottom row were textured with the non-BTF version of the same materials. Considerable difference in the appearance realism is evident.
At first sight there is noticeable that the restriction to smooth textures, contrary to BTF textures, does not allow realistic visual effects like reflections or colour changes due to varying illumination and viewing conditions. The example of utilization of the BTF texture measurements in the interior design is shown in the Figs. 16, 17. To improve the performance of BTF texture manipulation, BTF Roller synthesis step has been implemented into texture plugin. This algorithm contemporary represents the fastest way to seamlessly generate BTF texture of an arbitrary size.
Usage of BTF textures in Blender does not significantly slow down the rendering process. The most time consuming part is loading of BTF image data for a desired combination of illumination and viewing direction. The optimized version, as described in section 4.2, reduces this rendering time to 10%.

VII. CONCLUSION
We presented the novel Blender plugin and the corresponding minor Blender modification which together enable physically correct realistic rendering of surface materials represented in their most advanced form the bidirectional texture function. This plugin produces from this open source graphical software system the only rendering system available which allows correct surface materials presentation.
Our current implementation works with sampling based method for BTF texture enlargement. However we plan to generalize the plugin also with some BTF mathematical models. This will not only substantially increase the BTF data compression rate but it will simultaneously allow also rendering of BTF edited measurements. Finally the future plugin release will benefit from the multithread functionality.