The invention relates to an augmented reality system and method. In particular, the present invention relates to an augmented reality system in which actions are taken in relation to computer generated objects in response to the failure to detect a marker associated with that object.

Background of the Invention

In this specification, where a document, act or item of knowledge is referred to or discussed, this reference or discussion is not an admission that the document, act or item of knowledge or any combination thereof was at the priority date:

(i) part of common general knowledge; or

(ii) known to be relevant to an attempt to solve any problem with which this specification is concerned.

Augmented reality ("AR") systems are computer generated systems that process both real world and computer-generated data. In particular, AR systems are able to blend (or "augment") real world video footage - captured by digital video cameras - with computer-generated graphical objects, that appear in real time in the captured video footage.

A classic example of an AR system is the "headup display" used in fighter aircraft, in which information about the aircraft's position and speed or regarding an enemy aircraft is displayed to the pilot superimposed over the real image of the pilot's view. The additional information is sourced from various locations including radar and data from the aircraft's object recognition systems. This overlaying of computer generated information onto real world information augments the pilot's knowledge and understanding of the field of view.

Some AR systems utilise objects in the video footage along with information about the relative angle and position of the camera used to capture the footage, in order to overlay graphical objects in a convenient location. For example, lines used on a tennis court or football field are used as references to overlay graphical objects on the video footage broadcast to viewers. Other AR systems rely entirely on the location of a detectable object such as a barcode or fiducial marker (see below) within the camera frame. For example, print advertisements sometimes include a barcode or fiducial marker, which, when viewed on a computer monitor using a PC camera and associated decoding software, displays a three-dimensional model of an object such as a new car. As the advertisement and associated barcode or fiducial marker rotates or moves, the corresponding computer graphic object of the car also moves, creating an impression of a real car that is viewable by a potential buyer from different angles.

AR thus provides a means of enhancing the physical world beyond normal human experience. Within this field there is a huge and growing range of applications, including the areas of education, games and entertainment.

As mentioned above, real world objects appearing in the footage may be tagged with markers, that are used by motion tracking algorithms as measures and as points of reference. Markers - which may be considered as AR objects in themselves - are known in the art as "fiducial markers" or "fiducials". Other types of markers used in AR systems include images and special codes, such as BCH codes and barcodes.

When a marker is detected, the graphical object associated with the relevant real world object is rendered at the current position of the marker, or at another point as determined by the underlying AR software. If a marker is obscured from view, such as by an errant finger in the video camera, or by another object appearing in the captured video scene, conventional current AR systems cease rendering the object. Accordingly, the object disappears from the AR display (which may be a computer monitor, video camera, mobile phone, headgear etc.) and is only rendered again when the marker is detected at a later time. Objects can then flicker in and out of existence on the display, producing an unrealistic and distracting effect.

In other instances, the object continues to be rendered onscreen even after the marker fails to be detected, in order to create a more realistic scene. When the marker is redetected, possibly in a new location, the object is re-rendered at its new location However, this implementation is problematic in the scenario where the marker has moved (possibly permanently) out of the camera's field of view, as opposed to where the marker continues to be within the field of view but nevertheless is not detected. Current systems continue to render the object at its last known location, which tends to result in the creation of a somewhat unrealistic scene. Objects can appear to jump from point to point on a monitor, or remain stationary when in fact the marker may have moved permanently out of the field of view.

The present invention aims to address or ameliorate this deficiency in current AR systems.

Summary of the Invention

According to a first aspect of the present invention there is provided a method for implementing an augmented reality system, the augmented reality system comprising one or more graphical objects, each having an associated marker, the method including the steps of: displaying a graphical object when its associated marker is detected by a detection means; and upon a failure to detect the marker, ceasing the display of the graphical object if the marker was last detected in a selected region of the field of view of the detection means, or continuing the display of the graphical object if the marker was last detected in another part of the field of view of the detection means apart from the selected region.

The present invention predicates the decision whether to continue or cease displaying a graphical object in an augmented reality system on the basis of the last known location of the object's associated marker. It has been realised that markers may fail to be detected for a number of different reasons and that the last known location of the marker can provide valuable information regarding why the marker was not detected.

Preferably, the selected region is an area within the field of view, in which a failure to detect the marker can be attributed to the marker moving out of the field of view. According to this embodiment, graphical objects continue to be displayed when it is more likely that the marker was obscured by another object in the field of view (such as a finger, hand or pencil) rather than the detection-failure resulting from the marker being physically moved outside of the field of view. Conversely, display of the graphical object is ceased when it is more likely that the detection- failure was due to the marker moving outside of the field of view. Realistic augmented reality scenes can be created through use of this embodiment of the invention.

Typically, the area of said selected region is the periphery of the field of view. When a marker fails to be detected and was last known to be located in the periphery of the field of view, it is likely that the detection-failure is a result of the marker moving outside of the field of view, rather than a result of the marker being obscured by another object in the field of view.

Preferably, the method includes the step of periodically flagging each marker with a value indicating whether the marker was detected in the selected region. According to this embodiment, upon a failure to detect a marker, the value is examined and the display of the associated object either ceased or continued accordingly.

Optionally, the method may include the step of transitioning the graphical object from a previously detected position to a new detected position in the display.

Transitioning may include applying an animation to the object, such that the object is continuously displayed between the previously detected position and the new position. Transitioning may also include moving the object a proportion of the distance between the previously detected position and the new detected position at a set distance per time unit.

Animations are typically applied on the basis of the object and may include walking, running, driving and the like.

According to a second aspect of the present invention there is provided an augmented reality system comprising: a processor; a camera coupled to the processor for capturing video within a field of view; a display coupled to the processor for displaying the captured video; and an augmented reality application adapted for execution on the processor, the application having: a detection routine for detecting one or more markers in the captured video and associating the marker with a graphical object displayable on the display; and a display routine configured to, upon a failure to detect a marker, cease display of a graphical object if its associated marker was last detected in a selected region of the field of view, or continue display of the graphical object, if its associated marker was last detected in another part of the field of view apart from the selected region.

Preferably, the selected region is an area within the field of view, in which a failure to detect the marker can be attributed to the marker moving out of the field of view.

Typically, the area of said selected region is the periphery of the field of view.

Preferably, the augmented reality application includes a flagging routine for periodically flagging the or each marker with a value indicating whether the marker was detected in the selected region. Upon a failure to detect a marker, the value is examined and the display of the associated object either ceased or continued accordingly. ^'

Optionally, the augmented reality application includes a transitioning routine for transitioning objects from a previously detected position to a new detected position.

The transitioning routine may be configured to apply an animation to the object, such that the object is continuously displayed between the previously detected position and the new position. Transitioning may also include moving the object a proportion of the distance between the previously detected position and the new detected position at a set distance per time unit. Animations are typically applied on the basis of the object and may include walking, running, driving and the like.

According to a third aspect of the invention there is provided a software product stored on a computer readable medium, the software product including computer program code which, when executed by a processor, is configured to execute the method according to the first aspect of the invention or implement the augmented reality system according to the second aspect of the invention.

Brief Description of the Drawings

An embodiment of the invention will now be described with reference to the accompanying drawings in which:

Figures 1 to 4 are screen shots of an AR system implemented in accordance with the present invention, showing a fiducial marker with its associated graphical object;

Figure 5 is a flow chart of the method steps for implementing an AR system in accordance with an embodiment of the present invention;

Figures 6 to 9 are schematics, illustrating alternative implementations of a peripheral region and an interior region of a field of view in an AR system in accordance with an embodiment of the present invention; and

Figures 10 to 13 are schematics, illustrating an AR system in accordance with an embodiment of the present invention

Detailed Description of the Drawings

The present invention is implemented within the context of an augmented reality (AR) system. As discussed above in the background to the invention, an AR system blends real world video footage captured by a detection means, such as a digital video camera, with computer-generated graphical objects. A number of software platforms are currently available for developing AR systems, such as the ARToolkit from Canterbury University, Christchurch, New Zealand and Studierstube from Graz University, Austria. These platforms provide APIs and library routines implementing basic AR functionalities, including image detection and fiducial marker recognition and tracking. As discussed above, markers may be images or special codes such as BCH codes or bar codes.

Turning to Figure 1, a scene created in an AR system and displayed on a PC screen is illustrated. A fiducial marker 10 is located on a surface in front of the PC and is detected by a digital video camera (not shown) attached to the PC. The camera captures video footage only within a bounded field of view 13. Field of view 13 is illustrated in Figures 1 to 4 as a lighter rectangular area within a larger, blackened rectangular area.

Fiducial marker 10 has an associated graphical object 12 that is generated by the software platform running on the PC and displayed on the PC screen blended with the video footage captured by the PC camera. Object 12 is displayed when its associated fiducial marker 10 is detected by the PC camera.

A fiducial marker may fail to be detected when it is obscured by another object in field of view 13 of the video camera. This is illustrated in Figure 2 by a hand 14 covering fiducial marker 10.

A fiducial marker may also fail to be detected when it moves completely out of field of view 13. In the former instance, it is preferable to continue displaying an associated graphical object, because the marker remains within the field of view and the disappearance of the marker is more likely to be temporary. Ceasing the display of an object each and every time its marker disappears from view, possibly for a limited time period, results in a somewhat unrealistic AR scene and an unsatisfactory user experience.

On the other hand, it is preferable to cease displaying a graphical object when its associated marker has moved completely out of the field of view. Continuing to display an object, possibly at the last known position of its associated marker, after the marker has moved out of the field of view, also results in an unrealistic AR scene and unsatisfactory user experience.

It is therefore apparent that different actions must be taken with regard to either continuing or ceasing the display of a graphical object upon the failure to detect the object's associated marker, depending on the reason for the failure to detect the marker. To investigate the reason for a failure to detect a marker, the inventors have divided field of view 13 by way of a rectangular boundary 16 into two different regions. A first region 18 is the area defined within boundary 16 and a second region 20 is the area defined outside boundary 16 up to the edge of field of view 13. Region 20 is marked as 'Dead zone' in Figures 1 to 4 and is the periphery of field of view 13.

As discussed in further detail below, second region 20 can be considered as a 'warning area' for markers and their associated graphical objects. In particular, the inventors have realised that when a marker is located in second region 20 and then subsequently fails to be detected, it is more likely that the detection-failure is due to the marker leaving field of view 13 than the marker being obscured by another object in field of view 13.

As illustrated in Figure 2, when marker 10 is located in region 18 and subsequently fails to be detected, the detection-failure is likely to result from marker 10 being obscured by another object in field of view 13, and so associated object 12 continues to be displayed notwithstanding the failure to detect marker 10.

Alternatively, as illustrated in Figures 3 and 4, when marker 10 is located in second region 20 and subsequently fails to be detected, the detection failure is likely to result from marker 10 moving out of field of view 13, and so the display of associated object 12 is ceased.

A flow chart of a method used to implement the AR system functionality is shown in Figure 5. The method steps are executed by a software product built on top of a suitable AR software platform, such as those discussed above. At step 30, all of the markers present in the AR scene (the camera's field of view) are detected, by way of suitable image detection/marker tracking algorithms. At step 32, a decision is made as to whether a particular marker is detected. Markers are detected when the tracking algorithm identifies at least one marker within the camera's view angle. Where a marker is detected, a pose estimation algorithm is deployed to determine the marker's position and orientation. This assists in determining whether the marker is in an acceptable (see below) area of the camera's view.

Markers according to the invention are created as data structures having a number of attributes. Each attribute may be assigned a selected value. In particular, an important marker attribute relates to whether the marker is a "removable" marker or a "non-removable" marker. An attribute is a variable whose value is varied in response to events occurring within the AR system (see below) .

At step 34, in the event that a marker is not detected, a determination is made as to whether the marker is removable. This is implemented simply by querying the relevant attribute value of the marker.

If the marker is set to removable, the object associated with the marker is not rendered on the AR display (step 36).

Alternatively, if the marker attribute is not set to removable (i.e. the marker is "non-removable"), the associated objected is rendered in the display in its last known position (step 38).

Rendering a graphic object associated with a non-removable marker, in the event of not detecting that marker, maintains a continuous and consistent rendering of the object. Rendered objects do not "jump" throughout the display and are not constantly rendered and deleted in an attempt to follow a fast moving and disappearing marker, as occurred in prior art AR systems.

The method returns to step 30, where markers are again detected, after either step 36 (where the object is not displayed) or step 38 (where the object is displayed in its last known position) .

At step 40, in the event that a marker is detected, a determination is made as to whether the object is in region 20 (i.e the "dead zone"). As discussed above, when markers move into the dead zone and subsequently fail to be detected, it is likely that the marker has moved out of field of view 13.

Other 'non-valid' locations can be considered dead zones, including locations at a depth considered prohibitive for correct tracking. In the event that a marker is detected in a dead zone, the relevant attribute value of the marker (at step 42) is set to 'removable'. It will be realised that the marker will not be displayed if it subsequently fails to be detected during later iterations of steρs 32 to 38. Alternatively, where a marker is detected outside the dead zone (i.e in region 18), the marker's attribute value (at step 44) is set to 'non-removable'. Similarly, in later iterations of steps 32 to 38, markers with 'non-removable' attributes continue to be displayed notwithstanding a failure to detect the marker.

In either case (i.e. irrespective of whether the marker is either inside or outside the dead zone) a decision is made at step 46 as to whether the marker is detected in a location different from the location in which it was previously rendered.

If the marker is detected at a different position then, at step 48, the associated object is animated and displayed at its new location. Objects are animated to their new position based on an algorithmic interpolation between the old and new location and/or orientation. Action is taken to ensure a seamless transition of the object to its new location. Usually, this is in the form of applying an appropriate animation (walking, running, driving, etc) from the old location to the new location.

Previously, objects were instantaneously removed and re-rendered in their new location, which, as discussed above, tended to create an unsatisfactory user experience.

Where a marker is detected at its last rendered location, at step 50 the associated object is again rendered at its current position.

At step 52, the most recent rendered position of the object is recorded for further processing.

To increase fluency and believability, 'exit animations' can be included, by which (if an object is to be removed from the display), it leaves the scene in a suitable manner (i.e. a graceful departure, running, driving, etc). A similar approach can be taken for introduction of objects newly detected, with objects animated from off-screen to a position corresponding to the first detected position of the marker. The process returns to step 30 for further detection of markers in the AR system. A dead zone for use in accordance with the invention can be marked using: pixels; or percentage width of the screen

As illustrated in Figures 6 and 7, a pixel marking does not vary with the screen resolution of the display, i.e. it is always a set number of pixels from the edge of the screen (For example 5ρx). If the screen resolution is 320x240, then the renderable area would be 310x230. (Figure 6). Alternatively, if the screen resolution is 640x480, then the render able zone would be 630x470. (Figure 7)

As illustrated in Figures 8 and 9, a percentage width-marking does vary with the screen resolution i.e. is a relative width from the edge of the screen (For example 10%). If the screen resolution is 320x240, then the render able zone would be 288x216 (Figure 8) If the screen resolution is 640x480, then the render able zone would be 576x432. (Figure 9).

An embodiment of the invention is further described with reference to Figures 10 to 13. The terms 'render' and 'display' are intended to be used interchangeably. As discussed above, the invention operates along the lines of:

(a) if a fiducial can be detected, render object;

(b) if the fiducial cannot be detected and its last known location is in the dead zone, then stop rendering the object; and (c) if the fiducial cannot be detected and its last know location does not fall in the dead zone, keep rendering the object on the last known location.

In Figures 10 to 13, "P" represents the fiducial, and the suffixed number represents its position (Pl being the first position, P2 being the second position and so on) . In Figure 10, the objects are rendered at Pl and P2, based on concept "a". In Figure 11, the objects are rendered at Pl and P2, based on concept "a".

In Figure 12 the objects are rendered at Pl, P2, P3, P4 as normal. P5 is not detected and the last known location P4 was in the dead zone, so stop rendering the object, based on concept "b". In Figure 13, the objects are rendered at Pl, P2 as normal. P3 is not detected and the last known location P2 was in the render able area, so continue rendering the object, based on concept "c"

The word 'comprising' and forms of the word 'comprising' as used in this description and in the claims do not limit the invention claimed to exclude any variants or additions.

Modifications and improvements to the invention will be readily apparent to those skilled in the art. Such modifications and improvements are intended to be within the scope of this invention.