Technical Field

[0001] This disclosure generally relates to the technical field of computer graphics. More specifically, this disclosure relates to optimising computer rendering of multiple visual effects.

Background

[0002] Rendering is the process of generating a two dimensional (2D) image for one or more objects by means of a computer program, such as a graphical engine. Graphical engines typically follow a “rendering pipeline” that functions to render a 2D image in relation to a virtual camera, 3D objects (with width, length and depth) and light sources.

[0003] 3D objects of graphical engines (also referred to simply as “objects”) have various visual attributes that characterise how they should appear visually. These visual attributes include the object’s colour, visibility, texture, and shader.

[0004] Once the central processing unit (CPU) has processed the visual attributes of an object, it is then sent to a graphical processing unit (GPU) to be drawn on a display by manipulating the pixels of a corresponding display. This drawing request sent from the CPU to the GPU is called a “draw call”. The computational cost of a draw call substantially depends on the changes in the settings of the GPU to render the draw call. As draw calls are made on objects with differing visual attributes, the GPU needs to change its settings to render the given object as the visual attributes are changed. Hence, when these GPU settings do not change between draw calls, the draw call is inexpensive and takes little time to execute, compared to when the settings do change. To reduce the computational costs of rendering, draw calls are grouped to be drawn together in a “batch”. By batching objects to be drawn together, the amount of state changes needed to draw each object inside the batch is minimized. As such, it is desirable for the draw calls to be grouped into the least number of batches (known as the “batch count”) required.

[0005] It is often desirable, and necessary, to change the visual appearance of objects when rendering in real-time. Most graphical engines allow changes to the various visual attributes of an object in real-time. When new visual effects are applied to an object, graphical engines typically store a back up of the previous visual attributes of the object in memory. This has two negative effects. It consumes the memory in the computer and increases the batch count that must be drawn for every frame, particularly when rendering in real-time. When a visual attribute of an object is changed, the previous visual attribute is stored so that the change can be reversed. However this causes the draw call to be batched separately to another object with an identical visual appearance. A high batch count means less draw calls are batched together, resulting in excess data transfer between the CPU and GPU.

[0006] Any discussion of documents, acts, materials, devices, articles or the like which has been included in the present specification is not to be taken as an admission that any or all of these matters form part of the prior art base or were common general knowledge in the field relevant to the present disclosure as it existed before the priority date of each of the appended claims.

[0007] Throughout this specification the word "comprise", or variations such as "comprises" or "comprising", will be understood to imply the inclusion of a stated element, integer or step, or group of elements, integers or steps, but not the exclusion of any other element, integer or step, or group of elements, integers or steps.

Summary

[0008] There is provided a method for applying one or more visual effects to a plurality of objects for rendering in a computer, wherein each visual effect is characterized by its priority and type, the method comprising: storing in memory a visual effect list , the visual effect list comprising, for each of the plurality of objects, a set of original visual effects having a priority and type that define the object's original appearance; receiving a request for applying one or more additional visual effects, each having a priority and type, on the plurality of objects for rendering; storing in memory, the one or more additional visual effects in the visual effect list; processing, for each of the plurality of objects, the visual effect list to determine visual effects that should be applied to the object, being applied visual effects; providing a rendering queue of the plurality of objects whereby objects are ordered in the rendering queue in batches according to the applied visual effects; and causing each of the plurality of objects to be rendered based on the rendering queue.

[0009] In one embodiment, the method further comprises: processing, for each of the plurality of objects, the request by: comparing the type of each additional visual effect to the type of each original visual effect associated with that object to determine whether there are conflicting visual effects; and, if there are conflicting visual effects and, if required, modifying the additional visual effect's priority, such that the additional visual effect has a different priority than the visual effect in the visual effect list. That is, the method may alter the priority of an additional visual effect, where appropriate, depending on the rules required by the system.

[0010] In one embodiment, the types of visual effects may include at least one of: visibility, colour, shader, texture, or material. It should be noted that a material visual effect type can describe one or more of the other visual effects. For example, if the material is "glass", then this will indicate, at least, a particular texture and shader.

[0011] In one embodiment, the request is received from, or generated by a graphical engine of a computer based on user inputs.

[0012] In one embodiment, the step of causing each of the plurality of objects to be rendered includes: creating and forwarding a draw call to a GPU of the computer.

[0013] In one embodiment, the draw call comprises commands executable by the GPU of the computer to render 2D images which are representative of the visual effects of the plurality of objects on a display.

[0014] In one embodiment, the objects with identical applied visual effects are placed in the same batch in the rendering queue.

[0015] In one embodiment, the step of generating the applied visual effects for each object further comprises prioritising higher priority visual effects from the visual effect list over lower priority visual effects of the same type. [0016] In one embodiment, the step of generating the applied visual effects further comprises prioritising visual effects by visual effect type, and prioritising visual effects of the same type by priority.

[0017] In one embodiment, prioritising the visual effects includes: sorting the conflicting visual effects by using type as a primary sort order and priority as a secondary sort order.

[0018] In one embodiment, the types of the visual effects are arranged such that a highest priority is assigned to visibility, and a lower priority is assigned to each of colour, shader, texture, or material.

[0019] In one embodiment, different visual effects of the same visual effect type are also arranged to have different levels of priorities, for example a red colour may be assigned a higher priority than a green/blue colour.

[0020] In one embodiment, the priorities of the visual effects are defined in the graphical engine and are determined based on their practical implications (e.g. visual effects representing hazardous conditions will be assigned higher priority levels).

[0021] In one embodiment, conflicting visual effects are visual effects which cannot, or should not be rendered simultaneously according to one or more rules configured in the graphical engine.

[0022] In one embodiment, the step of generating the applied visual effect may further comprise combining the one or more additional visual effects and/or original visual effects to form a blended visual effect, and including the blended visual effect into the visual effect list.

[0023] According to a second aspect there is provided a non-transitory computer readable medium comprising instructions stored thereon that, when executed by a processor, cause the processor to perform one of the above methods.

[0024] According to a third aspect, there is provided a system for applying one or more visual effects to a plurality of objects for rendering on a display of a computer, wherein each visual effect is characterized by its priority and type, the system comprising a processor configured to: provide a visual effect list, the visual effect list comprising, for each of the plurality of objects, a set of original visual effects having a priority and type that define the object's original appearance; receive a request for applying one or more additional visual effects, each having a priority and type, on the plurality of objects for rendering; add the one or more additional visual effects in the visual effect list; determine from the visual effect list, for each of the plurality of objects, applied visual effects that should be applied to the object, being applied visual effects; provide a rendering queue of the plurality of objects whereby objects are ordered in the rendering queue in batches according to the applied visual effects; and cause each of the plurality of objects to be rendered based on the rendering queue.

Brief Description of Drawings

[0025] Examples of the present disclosure will be described with reference to:

[0026] Fig. 1 shows a schematic diagram of a computer rendering system according to one embodiment of the present disclosure;

[0027] Fig. 2 shows a flow chart of a method for applying one or more visual effects to a plurality of objects for rendering, according to one embodiment;

[0028] Fig. 3 illustrates examples of visual effect lists;

[0029] Fig. 4 demonstrates how the number of draw calls is reduced according to an embodiment of the present disclosure.

Description of Embodiments

[0030] Computer-based visualisation systems are widely used for many different applications such as industrial monitoring facilities, entertainment (for example, video games), design, and similar thereof. Most graphical engines allow changes to various attributes of a 3D object, including its base colour, visibility (if you can see it), texture (the detailed ‘skin’ wrapped around the object, such as the grain in wood, facial features on a human, colouring on a cat etc.), shader (how it responds to and emits light) and material (a combination of all of the above - a material determines shader, base colour and one or more textures, although not all graphics engines support materials).

[0031] In one form, a graphical engine is a program executable on a CPU of a computer, and includes a set of preconfigured rules to determine how an object should be displayed. To change visual appearances of objects or scenes displayed on a screen of the computer, the graphical engine is required to convert information representing visual appearances of the objects or scenes, into commands which are executable by a GPU of the computer, to cause corresponding pixels of the display to change their state and status, thereby reflecting the desired visual appearances of the objects or scenes.

[0032] In a simplest form, an object is a graphical representation, typically of a three dimensional shape or structure but can also include two dimensional shapes, to which a set of visual attributes can be associated. The visual attributes control how the object is intended to be displayed.

[0033] Typically, visual attributes are changed in the order in which they are received by the graphical engine, such that the objects appearance will reflect the most recent effect applied to the object. For example, if a request to change an object's colour to red (from the original white) is received, the object will be rendered to appear red on the display. If a subsequent request to change the colour to green is received, the rendering of the object will appear green on the display. However, difficulties arise when a request to remove a colour is received. Continuing the example above, if a request to remove the red effect is made, the object reverts to white rather than green. If the green effect is then removed, the object will be rendered in red even though there is currently no request to render the object in red.

[0034] The issue described occurs because when a new effect is requested, the current effects are backed up and are restored if the new effect is later removed. Thus, if the effects are not removed in the reverse order from which they are requested, then the effects applied to the rendered object may not be sensible.

[0035] In addition, in some circumstances, it may not be desirable to simply apply effects in the order in which they are requested by the graphical engine. For example, in an industrial monitoring system, the colour red may indicate a hazardous state of a piece of equipment. For safety reasons, it may be desirable to continue to render the object as red even if requests to change the colour are received based on other operational data.

[0036] Furthermore, in many graphics engines, making changes to an object’s visual characteristics causes new materials to be created in memory. In one form, a 'material' includes a combined list of visual effects including visibility, colour, shader and texture of that object. That is, a new distinct material is created each time a request to change a visual characteristic is made, even if the final visual effect of a material is identical to another already being used. Therefore, some of the materials are redundant for all intents and purposes.

[0037] This has two negative effects - it consumes memory in the computer and it increases the number of ‘batches’ that must be drawn for every frame (known as the ‘Batch Count’), particularly when rendering in real-time. Each of these factors contribute to the performance of the computer, with batch-count being particularly significant. As industrial applications tend to have large amounts of repetition, it can be extremely beneficial to have as many objects as possible sharing a single material so that they can take advantage of dynamic batching (a mechanism where objects that share material and lighting may be drawn together, preventing excess data transfer between the CPU and GPU on the computer).

Overview

[0038] Fig. 1 illustrates a computer rendering system 100, also referred to as Runtime Effect Conflict Resolution system (a term which also describes the method performed by system 100), comprising a central processing unit (CPU) 102, a graphics processing unit (GPU) 104, a data storage 106, a display 108 and a bus 110. The bus 110 is a series of electrical connections that transmits data between the CPU 102, the GPU 104, the data storage 106 and the display 108. In some embodiments (not shown in Fig. 1), the GPU 104 may be integrated with the CPU 102 and directly communicate with each other without the bus 110.

[0039] The data storage 106 stores computer executable code 112, that, when executed by system 100, causes the computer rendering system 100 to perform the method 200 of Fig. 2 and thereby apply one or more visual effects to a plurality of objects for rendering on the display 108. The plurality of objects, and their visual effects, form a scene which mimics how the plurality of objects should appear in 3D (or 2D). Each visual effect is characterized by a priority and a type. [0040] Now with reference to Figure 2, initially, at step 202, a visual effect list is stored in memory. The visual effect list comprises, for each of the plurality of objects, a set of visual effects associated with that object that define its original appearance. The visual effects may comprise one or more of visibility, colour, shader, texture or material (which is a combination of visibility, colour, shader, and texture).

[0041] At step 204, a request for applying one or more additional visual effects on the plurality of objects for rendering is received. In one form, the request is generated by the graphical engine, and may be triggered by status changes of the one or more objects. For example, when implemented in an industrial monitoring system, and if a hazardous event associated with a monitored equipment has been identified, a request may be generated to indicate an additional visual effect should be applied to that equipment to indicate the hazardous event.

[0042] At step 206, the request, for each of the plurality of objects, is processed. The processing comprises sub-steps 206-1 and 206-2. At sub-step 206-1, the type of each additional visual effect is compared to the type of each visual effect associated with that object in the visual effect list to determine whether there are conflicting visual effects. Visual effects conflict when they cannot be rendered simultaneously, such as two different colours.

[0043] At sub-step 206-2, if there are conflicting visual effects, the visual effects are prioritised by their priority in the visual effect list.

[0044] The computer rendering system 100 then performs step 208 by storing in memory, the additional visual effects in the visual effect list.

[0045] At step 210, the visual effect list is processed for each object to determine which visual effects which will be actually applied to the object, those that are applied being the applied visual effects. In addition the objects are then grouped in batches in a render queue according to the applied visual effects. In one form, the batches are where objects have identical applied visual effects. Alternatively, the batches are where objects have particular visual effects which are identical, for example, if multiple objects are of the same colour.

[0046] At step 212, each of the plurality of objects is rendered based on the render queue. [0047] Consider the example illustrated in Fig. 3 where each object 302 in the 3D scene has an in-memory or on-storage visual effect list 304 of its current active effects. The visual effect list, in practice, may be stored independently on each object, such that each object has an object visual effect list, or all visual effects may be located in a central list. Where there is an object visual effect list is stored independently on each object, the aggregation of each object visual effect list is the visual effect list.

[0048] Each visual effect that is applied to an object has a type 306 and a priority 308, which are used to determine what part of the object's appearance changes in each effect, and the order in which the effects should be applied. A higher priority effect should always dominate a lower priority effect, except optionally for colour and texture, which are described in more detail below.

[0049] Whenever an additional effect is applied or an old effect is removed from an object, the list is re-processed and the visual effect list re-processed to determine which of the objects associated visual effects are applied visual effects, being those visual effects which are actively used on the object.

[0050] The first step in processing the visual effect list for a given object is sorting the effects into a predictable order based on their priority and type. Although there are a range of orders that could be used, these are identified as some of the most effective.

[0051] Initially, the visual effect list is sorted based on priority and type as in list 310, with effects relating to visibility appearing first in the sort order, followed by colour (red, green), texture (not shown in Fig 3), shader (glowing) and material (glass). By using type as the primary sort order and priority as the second, an object's colour or texture request will be applied even if shader and material effects which have a higher priority value exist, which may be ideal for situations where you want informational colouring or texturing to survive other major changes in appearance. In these cases, a rule saying “The engine on the car should be red if it is running” would ensure that the engine remained red, even if other rules tried to make the engine translucent through a higher-priority change in material.

[0052] Alternatively, the visual effect list is sorted based on type-weighted priority, where visible items appear first, but all other items appear in priority order with a secondary order of type, following the same order as described above (colour, texture, shader, material). With this sorting, colour options will NOT override a material that has higher priority as indicated in list 312. This is more suited for when major changes in appearance should mask the information you are trying to deliver through colour. In the example of the engine, making the object translucent with a higher-priority material would mask the colour completely.

[0053] If an object has not had visual effects applied before, a backup 314 should be made of its base visual effects. This backup usually appears as a layer with the minimum priority level. This ensures that all other visual effects are applied ‘on top’ of the original base appearance. This also allows us to restore the object to its original visual state when no other effects are present.

[0054] To improve efficiency, visibility rules 316 should be processed first. If an object is not going to be visible, there is no need to process any other rules to determine appearance.

[0055] Following that, visual effects should be processed so that the effects with the highest priority (those that are considered more important) are those that are applied, and that those with lower priority are either ignored or blended. Where effects share a priority, they have the following priorities - texture/colour, followed by shader and material as shown in list section 318. As some attributes of shaders and materials can include values of colour or texture, this ordering ensures that the colour and texture effects do not get overwritten by those present in the shader or material. This translates into a rough rule of ‘the more specific a rule, the more dominant it is, even if the priority level is the same’, which is similar to the fundamental rules of the CSS web standard.

[0056] In some situations, it may be desirable to blend effects instead of completely ignoring effects with lower priority. For example if two colours appear in the list of effects for an object, it is possible to blend the colours rather than simply use that of the higher priority. Blending may be simply a 50% mixture of each, may be weighted by priority, or may be resolved using other techniques such as vertex colouring. In its simplest form, vertex colouring is a way of colouring vertices of an object such that no two adjacent vertices are of the same colour.

[0057] If processing the visual effect list results in the creation of a different object material, typically being a combination of the colour, texture and shader effects that are to be applied, to the one previously in use for that object, the previously used material should be destroyed - unless it is in use elsewhere or is the backup 314. Note that even graphics engines with automatic ‘garbage collection’ such as Unity (a development platform) will not destroy static object materials, therefore this embodiment must take care to destroy them, if required.

[0058] If the visual effect list has been reduced so that only the original ‘backup’ effect 314 remains, the visual effect list can be destroyed or deleted to conserve memory and the original visual effects restored to the object.

[0059] As the last step of the process, objects 401 should be placed in a rendering queue as illustrated in Fig. 4 where rendering queue 402 is the rendering order before ordering and rendering queue 404 is after reordering. In this example, objects have the same applied visual effects are grouped together in the rendering queue to form a batch of objects having the same "object material" (object material in this context describing all of the visual effects which are applied to the object). If the graphical engine supports names for each object material, a hash of the object material name could be used to provide a code in the rendering queue for the object relating to the object material. Ultimately, reordering the rendering queue in this manner ensures that objects having the same object material are rendered together, which radically reduces the number of draw calls 406 when using an engine that supports Dynamic Batching. In the example shown in Figure 4, before the list 402 was ordered, a total of 16 draw calls are required to be sent to the GPU. In comparison, after reordering as shown in list 404, a total of 4 draw calls are required. Note in this case it has reduced draw calls by 75%. This is important in many industrial applications. As mentioned above, industrial visualisation applications tend to have large amounts of repetition, due to the fact that sites often have many of the same items (graphically displayed as objects) deployed on-site. For this reason, traditional draw-ordering (where you draw the object closest to the camera first, known as ‘painters algorithm’) tends to provide reduced overdraw, but a large number of draw calls. It has been found that the performance impact of overdraw is minimal - particularly on mobile applications - but the reducing the number of draw calls can create significant performance benefits. For example, in some virtual reality applications an un-optimised rendering queue results in only 1 frame per second being displayed by the device with an optimised rendering queue, using embodiments of the invention, resulted in 39 frames per second in the same hardware.

[0060] Objects that are not in their default state may be placed in the render queue with other objects sharing the same base material and set of visual effects. For example, if rendering cars, rather than rendering all cars together regardless of their colour (which will cause many batches to be generated), you can alter the rendering queue so that all cars with the same colour are drawn together, before other colours are applied (for example, orange cars are drawn together, followed by blue, then red).

[0061] The system includes functionality to allow late-loading (or streaming) of content to effect the original 314 texture or colour of the mesh. When these new original visual appearance options are set, the effects on the object need to be re-processed so that the new colour and/or texture can be seen.

[0062] The Runtime Effect Conflict Resolution system must also integrate with any other optimisation and effect systems present in the application, such as the level-of-detail system. Level-of-detail systems (called LOD systems) routinely make small objects invisible or less detailed at long distances. As such, the Runtime Effect Conflict Resolution system will either need to be aware of the actions of the LOD system to ensure it doesn’t override them, or the LOD system will need to be modified to hide objects via the Runtime Effect Conflict Resolution system - otherwise the two systems might fight for control of object visibility, with LOD turning objects off and Runtime Effect Conflict Resolution turning objects back on.

Example application

[0063] Consider an example application involving a computer system showing the live status of 8 different vehicles. The engines of each vehicle currently share the same material (visibility, colour, texture, shader), meaning they are drawn in one batch.

[0064] The engine in each of the cars normally looks ‘real’ - it has a material that includes a colour (white), a texture (including the colours of the engine including hoses, caps and other components), and a ‘metallic’ shader ensuring it shines appropriately under lighting.

[0065] The user sets a first rule to tint the object blue when the engine is off. This effect is given a priority of 0.

[0066] The user has also chosen to tint the object green when the engine is on. This effect is also given a priority of 0, as it should never conflict with the first rule.

[0067] Finally, the user has also chosen to tint the object red when it is overheating. This effect is given a priority of 10, as it should override the other colours. In this way, the more important information - the safety-critical temperature alarm - will be visible over the less-important running status information.

[0068] When the system first opens, the engine is not running. The Runtime Effect Conflict Resolution system has only one active rule, to turn the engine blue.

[0069] First, the list of visual effects is sorted by priority and type as described above in relation to Figures 2 and 3.

[0070] Next, as this is the first time any form of visual effect has been applied to the engine, a backup 314 of its existing appearance (material) is taken and placed into the visual effect list with priority 0.

[0071] There are currently no rules governing the visibility of the object, so visibility status need not be considered.

[0072] Effects are then processed in order. In this case, the active effect (Colour: Blue) is applied to the material at priority zero. This results in the creation of a new material, as there are no other blue engines in the scene. This object will be drawn in its own batch. Note that even though the priority zero backup material has a ‘white’ colour, the fact that we sort our rules visibility/colour/texture/shader/material means that our colour is not overwritten by the material settings - this is because the material and the colour are merged together.

[0073] As the above step has resulted in the creation of a new material, the system checks if the previously used material should be destroyed. In this case, the material we are replacing is the backup material, so it should not be destroyed.

[0074] The object is now placed into the rendering queue based on its unique material ID, plus a modifier based on its colour (blue). When the other engines are processed in a similar fashion, this results in all stopped engines rendering in the same draw call, reducing from 8 potential batches, one batch for each vehicle, to a single batch. There are 2 materials in memory (the original and the current visual effects).

[0075] The engine on car #1 now changes to ‘starting’. There is no specific colour rule for the engine being in the ignition phase, so the blue colour is removed. [0076] First, the list of effects is sorted, containing a single item, which is the colour of car #1. No backup is needed, as it already exists in the list. The backup material is applied to the engine and the entire effect list is destroyed, as the object is now back to its original appearance and position in the render order. There are currently 2 batches - one for the blue engines, one for the engine in its natural state. There are only 2 materials in memory.

[0077] The engine on car #1 is now ‘running’. This adds the green tint to the list of effects. The list is sorted by priority and type, a backup is created and effects are processed in order. This results in two batches again - one for the blue engines, one for the green one.

[0078] The user of the system chooses to turn on their X-Ray vision so that they can view information about the suspension system. This causes a new material to be added to the effect list for every engine - this material offers a new colour (grey), texture (non) and shader (semitransparent) for our object and has a priority of 5.

[0079] The list of effects are sorted, resulting in the X-Ray Vision material being first in the sorted list as it relates to visibility. That material is applied to the engines. As there are no other objects in the scene with running engines, the green material previously being used is destroyed. The engines lose their metallic shine and texture and become transparent grey. There are now still only 2 batches and 2 materials in memory (Original & X-Ray Vision).

[0080] Engine of car #1 now begins to overheat. This results in the effect to turn the engine red being turned on.

[0081] When sorted, the colour red is found first in the sorted visual effect list due to its priority of 10. After that is the X-Ray Vision material. The red colour is merged with the X-Ray Vision material to change the material from translucent grey to solid red. They still do not have their metallic shine or texture, as those were on the original, backup material and have been overridden by X-Ray Vision. The plain X-Ray Vision material is not destroyed, as it is present on each of the other engines, and it is only engine #1 that is overheating. Now we have 2 batches and 3 materials in memory (Original, X-Ray Vision & Red X-Ray Vision) with 2 of them being used. Engine #1 now has the following items in its visual effects list (in order) - Red (colour), X- Ray Vision (material), Green (colour) & Backup (material). [0082] The user then turns their X-Ray Vision mode off. This removes the X-Ray vision effect from all objects in the scene. For car #1, this results in the colour red being found first, followed by the backup material. The red colour is merged with the objects original material, so we now see a red-tinted engine that once again has its metallic shine and texture. We return to 2 batches and 2 materials in memory.

[0083] The user then moves the virtual camera a significant distance from the vehicles. The level-of-detail (LOD) system chooses to hide the object and does so via the Runtime Effect Conflict Resolution system, adding a new ‘Hide’ rule with a priority of 200.

[0084] When sorted, the ‘hide’ rule is found at the top of the visual effect list. Processing terminates immediately and the object is hidden.

[0085] While hidden, the overheat ceases and the engine returns to normal temperature. The ‘red’ rule is removed. The ‘green’ rule remains. The engine itself remains unchanged due to the fact that its hidden.

[0086] The virtual camera now moves closer again, and the level-of-detail system removes the ‘hide’ rule. When the effects are sorted and processed, a green engine is displayed, indicating that it is running.

[0087] Throughout the process, the Runtime Effect Conflict Resolution system ensured that the data represented by colour is still accurate even though changes occurred at a time where they couldn’t be made visible or were overridden by more important information. Furthermore, the appearance data was restored even though effects were applied and removed out-of-sequence. In addition, the number of materials and batches is kept to a minimum, maximising performance of the computer system.

[0088] It will be appreciated by persons skilled in the art that numerous variations and/or modifications may be made to the above-described embodiments, without departing from the broad general scope of the present disclosure. The present embodiments are, therefore, to be considered in all respects as illustrative and not restrictive.