METHOD AND APPARATUS FOR GRAPHICALLY DEFINING

SURFACE NORMAL MAPS

CROSS-REFERENCES TO RELATED APPLICATIONS [0001] This application claims the benefit of U.S. Provisional Application No. 60/953,665, filed August 2, 2007, entitled "Method and Apparatus for Graphically Defining Surface Normal Maps," and of U.S. Patent Application No. 12/072,665, filed February 26, 2008, entitled "Method and Apparatus For Graphically Defining Surface Normal Maps".

BACKGROUND OF THE INVENTION

[0002] The present invention relates in general to computer-generated animation and in particular to graphically defining surface normal maps.

[0003] Three-dimensional (3-D) animation generally begins with a geometric model of the objects that will appear in the animated scene. Each object is modeled, e.g., as a mesh of polygons in 3-D space, and various attributes of the object's surface are associated with points in the mesh, such as the vertices of the polygons. For example, attributes associated with a point often include a color, a surface normal, a transparency parameter, reflectivity parameters, and one or more sets of texture coordinates, allowing one or more textures to be applied to the surface. [0004] To generate (render) the images, the positions of various objects in the scene are established; for animated images, each image is generated to correspond to a particular time, and positions of at least some objects may vary with time. A viewpoint, or virtual camera position, is established, and a screen area (generally normal to the camera) is defined. The screen area is divided into small sub-areas, referred to herein as pixels, and a color for each pixel is determined based on the attributes of the object (or objects) that project onto that pixel. Which object(s) project onto a pixel can be determined using a variety of techniques, including ray- tracing. In ray tracing, rays are drawn from the pixel to the object (or from the object to the pixel), and the intersection of the ray with the object's surface determines which portion of the object's surface (e.g., which polygon or which vertices) should be used to compute the pixel's color. Computers are used extensively in both the modeling and rendering phases.

[0005] Computer-generated 3-D animation (referred to herein as "CGA") usually approximates a photorealistic look. Objects have crisp, smooth edges and surfaces that do not bleed or smear into each other. In fact, one of the problems CGA faces is that surfaces and edges often look too smooth, lacking the roughness and imperfections of real-life objects. [0006] Further, the photorealistic look of CGA is esthetically limiting. Traditional hand-drawn animation allows the animator to depart from a photorealistic look and adopt a more "painterly" style, with uneven brush strokes, "loose" paint at edges of objects and so on. The traditional animator can adapt the look of the animated world to fit the story being told, and this stylization is generally regarded as one of the advantages of animation over live action.

[0007] Efforts to duplicate this painterly look in CGA have not been satisfying. For instance, paintbrush textures have been applied to rendered scenes, but the result is usually a displeasing "screen door" effect as the characters and other objects move under a fixed texture. Other attempts to apply paintbrush-like textures to objects have led to distracting "popping" as loose fragments of virtual "paint" appear and disappear from one frame to the next. Some techniques for incorporating painterly elements, e.g., into backgrounds, have been developed, but these techniques generally have not scaled well or been easy to integrate into CGA processes.

[0008] It would therefore be desirable to provide improved computer-based techniques for rendering images with a painterly look.

BRIEF SUMMARY OF THE INVENTION

[0009] In a painting, the surfaces of the objects depicted tend to include visible artifacts of the brushstrokes used to create them. For instance, edge and interior lines are generally not perfectly straight or uniform in width, curved surfaces do not look perfectly smooth or uniform, and so on. In computer generated images, such non-uniformities can be recreated by perturbing surface normal maps, but it is difficult to tell from a surface normal map what the ultimate effect of a given perturbation will be.

[0010] Accordingly some embodiments of the present invention provide visualization tools that facilitate a user's understanding of surface normal maps and the effect of making a change to a surface normal map. An intuitive visual interface allows a user (e.g., an art

director, although a user could be anyone participating in the definition of an object to appear in a computer-generated image) to readily achieve a desired painterly style.

[0011] In one embodiment, the surface normal components on the x, y, and z axes (in object coordinate space) are mapped onto red, green, and blue color components, respectively. (Other mappings of surface normal components to color components may be substituted). The object's surface is displayed in an editing interface with colors indicating the surface normals. Through a graphical interface, the user can then modify the surface normals to achieve a desired look for the object. The modified surface normals are stored with the object-model data and are used in rendering the object, giving the object a painted appearance.

[0012] The following detailed description together with the accompanying drawings will provide a better understanding of the nature and advantages of the present invention.

BRIEF DESCRIPTION OF THE DRAWINGS [0013] FIG. 1 illustrates a reference object with surface normals represented as a color map according to an embodiment of the present invention.

[0014] FIG. 2 shows the reference object of FIG. 1 after a paintbrush-like perturbation has been applied to the RGB color components of the remapped surface normals.

[0015] FIG. 3 is a rendered image of the object of FIG. 2 after converting the perturbed RGB color component map back to a surface normal map according to an embodiment of the present invention.

[0016] FIG. 4 is a flow diagram of a process for modeling an object according to an embodiment of the present invention.

[0017] It is noted that certain of the drawings are renderings best viewed in color. Color drawings are expected to be available via the U.S. Patent and Trademark Office, with specific reference to U.S. Patent Application No. 12,072,665.

DETAILED DESCRIPTION OF THE INVENTION

[0018] FIG. 1 illustrates a reference object 100, a cylinder 102 with flanges 104, for which surface normals are represented using a color map according to an embodiment of the present

invention. In this embodiment, reference object 100 is defined in an (x, y, z) coordinate space (e.g., object space) with the axis of cylinder 102 aligned with the z axis and they axis oriented vertically.

[0019] To aid in visualizing the surface normals, the x component of the surface noπnal at each point is mapped to a red color component (with higher red values reflecting larger x components), the y component to a green color component, and the z component to a blue color component. This mapping can be performed, e.g., in a paint program that allows a user (e.g., an art director) to define surface features of an object during modeling. Paint programs are known in the art. Such programs allow the user to view an image of an object, where the image is generated from the object model. A graphical user interface allows the user to rotate, zoom, and pan to see other views of the object. Further controls allow the user to modify aspects of the object model, e.g., reshaping the surface, adding additional elements, defining surface colors, etc.

[0020] In accordance with an embodiment of the present invention, the paint program can be configured with a "show surface normals" mode that maps the surface normal components to color components and uses the mapped color components to determine the color of the surface to be displayed. In this mode, color attributes defined by the object model are advantageously ignored; the color is determined using only surface normals. Thus, in "show surface normals" mode, the displayed image represents the direction of the surface normals and the variation from one portion of the object to the next.

[0021] For example, FIG. 1 shows an image that can be generated in "show surface normals" mode according to an embodiment of the present invention. In this embodiment, the x axis of an object coordinate space is mapped to the red color component, they axis to green, and the z axis to blue. If the object model defines the surface normals as normalized vectors in object coordinate space, the mapping can be accomplished by converting each vector component (which would be in the range from -1.0 to 1.0 for a normalized vector) to an intensity value for the corresponding color component in the range between a minimum value and a maximum value (e.g., 0 to 255). Linear mapping can be used for the conversion. In some embodiments, areas near sharp edges (discontinuities in the surface normal components) can be colored to reflect a "half angle" — a color halfway between the colors associated with the surface normals on either side of the edge.

[0022) It is to be understood that the "show surface normals" mode is simply a visualization tool for of surface normals and in particular for visualizing perturbations of surface normals; in other display modes of a paint program or during actual rendering, a different surface color map can be used. [0023] As can be seen in FIG. 1 , reference object 100 has a smooth appearance, with sharp, clean edges and corners at the flanges and cylinder ends, and looks photorealistic rather than painted.

[0024] In accordance with an embodiment of the present invention, the paint program allows a user to manipulate the surface normals by manipulating the RGB color of the pixels shown in the "show surface normals mode." In particular, the user can apply a paintbrush-like perturbation to create a more painterly appearance, with "loose" internal edges and/or corners, unevenness in curved surfaces, etc.

[0025] The surface-normal manipulation interface can be implemented in various ways. In one embodiment, "tumble and paint" interface similar to existing interfaces for manipulating images is used. The interface includes tumble tools that allow the user to modify the view of the 3-D object, so that the user can view the object from any angle, zoom in or out, etc. The interface also includes paint tools that allow the user to modify the color of the surface or a portion thereof. The paint tools advantageously provide a selection tool operable to select a portion of the surface to be modified, as well as various tools to perform modifications on the selected portion of the surface. Paint tools can include an expression-based texture manipulation tool that supports such operations as applying noise, remapping colors, warps and so on. Paint tools can also include a brush tool usable with a tablet input device or the like that supports painting or blending user-selected colors onto a surface; painted textures can be projected back onto the surface and stored as texture maps using conventional techniques. More generally, any paint tool that allows a user to modify the coloration of a surface can be used.

[0026] In some embodiments, the interface also provides a preview mode, in which the artist can see the object rendered with the modified surface normals. This mode allows the user to see directly the effect of the surface normal manipulations on lighting of the object. [0027] Thus, the artist can begin with an image colored according to the initial surface normal map and can select and blend between colors to make adjustments. The user can then

operate the tumble tool to change to a new view of the surface and continue painting. This process can be repeated until the user is satisfied.

[0028] Because it is color that changes as the user manipulates the object, the effect of the change is readily apparent. For example, FIG. 2 shows reference object 100', which corresponds to reference object 100 of FIG. 1 except that a paintbrush-like perturbation has been applied to the RGB color components according to an embodiment of the present invention. The appearance of the object in FIG. 2 is decidedly more painterly than that in FIG. 1.

[0029] Once the user is satisfied that the desired style has been achieved for the object, the modified RGB color components can be used to create a new surface normal map, e.g., by mapping the R, G, and B color components back to x, y and z components of a surface normal vector at a number of different points on the surface. (The inverse of the mapping transformation that was employed to generate the image shown in FIG. 1 can be used here.) The new surface normal map is stored with the rest of the geometry model for the object. [0030] The object can then be rendered using conventional techniques. FIG. 3 is an image of a reference object 100". The image in FIG. 3 is generated by converting the modified RGB surface normal map of FIG. 2 back to surface normal components, then rendering surface lighting using the surface normal map. Although surface coloring has not been applied, the shading of object 100" does reflect highlights and shadows rendered in accordance with the modified surface normal map of FIG. 2. As FIG. 3 shows, the surface of rendered object 100" takes on a painterly quality as a result of the modifications to the surface normal map.

[0031] FIG. 4 is a flow diagram of a process 400 for modeling an object with painterly surface normals according to an embodiment of the present invention. Process 400 can be performed, e.g., using a paint program of generally conventional design augmented with additional capabilities related to surface normal mapping as described herein. At step 402, an object model is defined. The object model includes a surface normal map from which a surface normal at any point on the object's surface can be determined. The surface normal map can be a high-resolution map as is commonly used for generating cinematic images. Conventional mapping techniques may be used to represent the surface normals as a function of position. For instance, a surface normal can be associated with each vertex of a polygon mesh, with surface normals for intermediate points being determined by interpolation.

Alternatively, a parametric mapping that defines surface normals as a function of (x, y, z) position in the object's coordinate space can be used. At this stage, the surface normals can be smoothly varying or otherwise appropriate for photorealistic objects.

[0032] At step 404, the surface normal map is remapped into RGB color space. Remapping can include, e.g., associating the (normalized) x, y and z components of the surface normal with red, green, and blue intensities, respectively, as described above. At step 406, an image of the object is displayed in the paint program, with the surface of the object being colored according to the RGB color map representing the surface normals, resulting in an image such as that shown in FIG. 1. This enables a user of the paint program to see the variation in surface normals and readily judge the smoothness and evenness of the surface.

[0033] At step 408, the user can modify the RGB color map. For example, the user can apply a perturbation to the colors that emulates the unevenness of a paintbrush stroke as shown in FIG. 2. The perturbation can be parametric, noise-based, based on predefined brush stroke patterns, or improvised by the user on the spot using the paint program to change pixel colors while viewing the image of the object until a desired quality of unevenness is achieved. The paint program's painting interface can be used to allow the user to alter colors for a pixel or swath of pixels. At step 410, when the user has finished modifying the RGB color map, the color map is converted back into a modified surface normal map. The inverse of the mapping at step 404 can be used. [0034] At step 412, the modified surface normal map is stored in association with the object. The object can then be rendered using the modified surface normal map. The modifications to the surface normals affect the lighting of the surface as is known in the art, and as a result, the object has a less perfectly round, more brush-stroked appearance, e.g., as shown in FIG. 3. In an alternative embodiment, the modified surface normal map can be used directly in image rendering rather than being stored first.

[0035] While the invention has been described with respect to specific embodiments, one skilled in the art will recognize that numerous modifications are possible. For example, the mapping of (x, y, z) to RGB color space could be changed, e.g., by mapping x, y, and z components to a different permutation of the red, green and blue color components, or by mapping into a different color space (e.g., the "negative" cyan, magenta, yellow space).

Other procedural perturbation techniques could be used to perturb or modify surface normals in addition to or instead of the specific techniques described herein. The processes and

techniques described herein can be used more generally to manipulate the surface normal map on a high-resolution object model to achieve treatments other than that shown, in order to achieve other particular artistic stylizations of the object.

[0036] It should be noted that surface normal perturbation as described herein will generally affect the appearance of non-silhouette portions of an object, in particular non-silhouette edges or curved surfaces. The surface normal perturbation generally has less effect on silhouette edges, where the surface normal is approximately at right angles to the viewer. This can be seen, e.g., in FIG. 3. It is to be understood that the present invention can be employed in conjunction with other techniques, such as the multiple-surface rendering techniques described in commonly-owned co-pending U.S. Provisional Patent Application

No. (Attorney Docket No. 026231-00231 OUS), filed of even date herewith, can be used in connection with the present invention to provide a more painterly look at or near silhouette edges. For example, multi-surface rendering can be used to create an appearance of loose paint at or near the silhouette edges. [0037] Some components of the processes described herein can be implemented using suitably-configured computer systems. Such systems may be of conventional design and may include standard components such as microprocessors, monitors, keyboards, mice, magnetic disk drives, CD or DVD drives, flash drives, network interface components, and the like. In addition, interconnected groups of computers (e.g., server farms) may be used to practice aspects of the present invention. While the embodiments described above may make reference to specific hardware and software components, those skilled in the art will appreciate that different combinations of hardware and/or software components may also be used and that particular operations described as being implemented in hardware might also be implemented in software or vice versa. [0038] Computer programs incorporating various features of the present invention may be encoded on various computer readable storage media; suitable media include magnetic disk or tape, optical storage media such as CD or DVD, flash memory, and the like. Such programs may also be encoded and transmitted using carrier signals adapted for transmission via wired, optical, and/or wireless networks conforming to a variety of protocols, including the Internet. Computer readable media encoded with the program code may be packaged with a compatible device or provided separately from other devices (e.g., via Internet download to a storage medium connected to the recipient's computer system).

[0039] Thus, although the invention has been described with respect to specific embodiments, it will be appreciated that the invention is intended to cover all modifications and equivalents within the scope of the following claims.