TECHNICAL FIELD

[0001] The present invention relates to a mixed reality visualisation system and, more particularly, in one form to such a system which permits accurate placement of constructs shown on a two-dimensional site map plan on the actual site depicted on the site map plan. In an alternative form the system permits construction of a 3D rendered construct from a 2D representation.

BACKGROUND

[0002] Mixed reality systems have begun to be used to assist users to better visualize how planned constructs may appear in reality.

[0003] Broadly the concept is to superpose a virtual depiction of the construct on a video view of reality. In some instances the reality may be a construction site upon which it is intended to build a construct in the form of a house.

[0004] A problem in implementation of such systems is inability to properly reference the virtual depiction of the construct against the reality.

[0005] A related and separate problem can be difficulty in seeking to properly reference in real time or near to real time .

[0006] A further related problem relates to constructing a virtual depiction in three dimensions from a two dimensional representation . [0007] It is an object of the present invention to address or at least ameliorate some of the above disadvantages or provide a useful alternative.

Notes

[0008] The term "comprising" (and grammatical variations thereof) is used in this specification in the inclusive sense of "having" or "including", and not in the exclusive sense of "consisting only of".

[0009] The above discussion of the prior art in the

Background of the invention, is not an admission that any information discussed therein is citable prior art or part of the common general knowledge of persons skilled in the art in any country.

SUMMARY OF INVENTION

Definitions

[00010] Mixed Reality: A mixed reality environment, also called an augmented reality environment, is one where an electronic image or video view of reality (for example, a user's present surrounds) has superposed onto it or integrated with it electronically generated imagery. In one form, the electronically generated imagery may represent a 3D rendering of a building.

[00011] Site: In this specification, a site is a location where a building may be constructed.

[00012] Floorplan: In this specification, a floorplan is the 2D plan, taken from above, of a building compromised of walls, floors, windows, doors, stairs and other. May also, but not necessarily contain elevation information which then in totality contains everything to construct a 3D model.

[00013] Siteplan: In this specification, a siteplan shows the 2D floorplan of the building to be constructed within the site boundaries and neighbouring sites and/or artefacts.

[00014] Artefact: In this specification, an artefact is a distinguishing feature found in the 2D siteplan that may or may not be also found in the live video at a site. Artefacts could be trees, walls, roads, utility poles (electricity, telephony) , fences, hedges, grass edges, planted gardens, gates, inclines, man holes, rivers/streams, zebra crossings.

[00015] 3D model: In this specification, a 3D model is the 3D model information comes from one of the following methods:

The floorplan contains top down views, side views and a scale therefore the 3D model can be constructed

The 3D model is supplied separately as a file containing the geometric information, by projecting this file in isometric projection (without perspective) from the top down the 2D floorplan is projected and this can be aligned with the existing 2D floorplan thereby aligning the 3D model with the 2D floorplan.

In one form it may be derived from the 2D floorplan where the missing height information is calculated as the default or defacto ceiling and window height for that region e.g.

220 cm for Australia. In this case the 3D model accurately represents the interior of the building but may not reflect exterior elements such as an angled roof.

Accordingly in one broad form there is disclosed a method for creation of a mixed reality 3-D experience from a two-dimensional representation of a three-dimensional structure when the three-dimensional structure is located in a predetermined environment; said method comprising selecting a predetermined environment; the environment referenced against a location defined against accordance in three dimensions;

selecting a two-dimensional representation for placement in the environment;

the two-dimensional representation having reference artefacts included in it which are located to scale relative to the two-dimensional representation;

constructing a three-dimensional representation from the two-dimensional representation;

placing the three-dimensional representation into a mixed reality environment;

in substantially real time projecting the three- dimensional representation visually to a user as the user navigates through the predetermined environment superposed with a reality vision of the predetermined environment.

Preferably the three-dimensional representation is referenced against the reference artefacts of the two- dimensional representation.

[00016] In a further broad form of the present invention there is provided a method for accurately aligning in 3D space a 3D model of a building on a live video view of a site using a 2D siteplan; said method including locating reference artefacts on a 2D representation of the building and site; utilizing the live video view to locate at least some of the reference artefacts on the site; anchoring the 3D model of the building against the at least some of the reference artefacts during the live video view thereby accurately aligning the 3D model of the building in 3D space with reference to the 2D siteplan . [00017] Preferably the site is the location where the building will be constructed.

[00018] Preferably the 2D site plan of the building

comprises a floorplan in the form of a 2D plan.

[00019] Preferably the 2D plan is taken from above of a building compromised of walls, floors, windows, doors, stairs and other.

[00020] Preferably the 2D plan contains elevation

information; the elevation information in totality contains data to construct a 3D model.

[00021] Preferably the 2D site plan comprises a 2D floorplan of the building.

[00022] Preferably the 2D floorplan of the building is constructed within site boundaries of the site.

[00023] Preferably the 2D floorplan of the building is constructed within neighbouring sites of the site.

[00024] Preferably the 2D floorplan of the building is constructed within one or more artefacts of the site.

[00025] Preferably each of the one or more artefacts is a distinguishing feature found in the 2D siteplan.

[00026] Preferably each of the one or more artefacts could be in the form of trees, walls, roads, utility poles

(electricity, telephony) , fences, hedges, grass edges, planted gardens, gates, inclines, man holes, rivers/streams, zebra crossings . [00027] Preferably the one or more artefacts also exist in the live video view of the site.

[00028] Preferably the one or more artefacts do not exist in the live video view of the site.

[00029] In yet another broad form of the present invention there is provided a method for accurately anchoring a

floorplan and/or 3D model to a live video view of a site on which the floorplan has been depicted on a site map; the method comprising producing an isolated site view; and wherein the isolated site view is produced by removing the floorplan from the site map by looking for floorplan artefacts such as the door and window geometry and then expanding to include geometry with straight lines forming concave shapes.

[00030] Preferably the live video view of the site comprises a video feed.

[00031] Preferably said video feed is produced by a camera in a phone, tablet or computer by the user (or electronics) .

[00032] Preferably said video feed is panning over a site area of the site detecting a horizontal plane of the site area .

[00033] Preferably said method detects artefacts on the horizontal plane of the site area.

[00034] Preferably the artefacts are detected from high contrast changes in video change that exists as the camera move; and wherein the high contrast changes are recognized as detected vertical objects; and wherein approximate shape and location of the detected vertical objects are concluded and calculated; and wherein the approximate shape and location of the detected vertical objects are included into a 2D view of the site map.

[00035] Preferably said method overlays the 2D view on the isolated plan view; wherein the method conducts a correlation by correlating the detected vertical objects of the 2D view with the artefacts of the isolated plan view.

[00036] Preferably the correlation is above a certain threshold, isolated plan view and the live video view of the site; and wherein the 3D model is projected to live video feed .

[00037] Preferably said method enables the user to walk through the site and the 3D model is accurately anchored to its designated position as defined by the live video view.

[00038] Preferably the one or more artefacts comprise a spacial coordinate.

[00039] Preferably the spacial coordinate is a location on the earth surface.

[00040] Preferably the spacial coordinate lies within a bounded planar surface to which the floor plan is to be applied.

[00041] Preferably overlays the 2D view on the isolated plan view; wherein the method conducts a correlation by correlating the detected vertical objects of the 2D view with the

artefacts of the isolated plan view.

[00042] Preferably the correlation is above a certain threshold, isolated plan view and the live video view of the site; and wherein the 3D model is projected to live video feed.

[00043] Preferably enables the user to walk through the site and the 3D model is accurately anchored to its designated position as defined by the live video view.

[00044] Preferably a siteplan may be electronically

generated from a floorplan and external image data.

[00045] Preferably the external image data is derived from satellite imagery referenced against GPS or other like spacial coordinates .

[00046] Preferably 2D overhead image of a site 210 together with at least one spacial coordinate 211 located within the site 210 is transmitted via internet 212 to an augmentation server 213.

[00047] Preferably the augmention server 213 receives a 2D plan 214 from an external source.

[00048] Preferably the 2D plan 214 may be a 2D plan 214 of a building structure.

[00049] Preferably the augmentation server 213 renders a 3D image of the 2D plan 214 and then locates the 3D image 215 with reference to the spacial coordinates 211 thereby to generate a mixed reality image.

[00050] Preferably the resulting mixed reality image 216 may then be viewed via an imaging device 217.

[00051] Preferably the imaging device 217 may be a digital communications device which includes a visual display 218 and radio communications capability 219.

[00052] Preferably the augmentation server is accessed via a web link on a website.

[00053] Preferably a 2D representation of a floorplan has heat map output superposed on the floorplan.

[00054] Preferably the heat map represents virtual person time spent at locations within the mixed reality image. [00055] In a further broad form of the invention there is provided a mixed reality headset having a display which combines reality vision of site of a pre-determined

environment with a three-dimensional representation of a three-dimensional structure; the representation of the three- dimensional structure referenced against an artefact in the pre-determined environment.

[00056] Preferably the site is the location where a building will be constructed.

[00057] Preferably the 2D site plan of the building

comprises a floorplan in the form of a 2D plan.

[00058] Preferably the 2D plan is a view taken from above of a building compromised of walls, floors, windows, doors, stairs and other.

[00059] Preferably the 2D plan contains elevation

information; the elevation information in totality contains data to construct a 3D model.

[00060] Preferably the 2D site plan comprises a 2D floorplan of the building.

[00061] Preferably the 2D floorplan of the building is constructed within site boundaries of the site.

[00062] Preferably the 2D floorplan of the building is constructed within neighbouring sites of the site.

[00063] Preferably the 2D floorplan of the building is constructed within one or more artefacts of the site.

[00064] Preferably each of the one or more artefacts is a distinguishing feature found in the 2D siteplan.

[00065] Preferably each of the one or more artefacts may be in the form of trees, walls, roads, utility poles

(electricity, telephony), fences, hedges, grass edges, planted gardens, gates, inclines, man holes, rivers/streams, zebra crossings .

[00066] Preferably the one or more artefacts also exist in the live video view of the site. [00067] Preferably the one or more artefacts do not exist in the live video view of the site.

[00068] Preferably the one or more artefacts comprise a spacial coordinate.

[00069] Preferably the spacial coordinate is a location on the earth surface.

[00070] Preferably the spacial coordinate lies within a bounded planar surface to which the floor plan is to be applied.

[00071] In yet a further broad form of the invention there is provided media incorporating code stored therein which, when executed by a processor effects the method described above .

BRIEF DESCRIPTION OF DRAWINGS

[00072] Embodiments of the present invention will now be described with reference to the accompanying drawings wherein:

[00073] Figure 1 is a schematic diagram illustrating a method for creation of a mixed reality 3-D experience from a two-dimensional representation of a three-dimensional

structure when the three-dimensional structure is located in a predetermined environment according to one preferred

embodiment of the present invention.

[00074] Figure 2 is a schematic diagram showing various channels to acquire at least 2D renderings from which to produce the three-dimensional structure according to any one of the preferred embodiments of the present invention.

[00075] Figures 3A and 3B are schematic diagrams

illustrating a method for accurately aligning in 3D space a 3D model of a building on a live video view of a site using a 2D siteplan according to one preferred embodiment of the present invention.

[00076] Figure 4 is a flowchart applicable to the

arrangement of Figure 3A, 3B.

[00077] Figure 5 is a schematic diagram of an alternative embodiment for creating a composite mixed reality image relying on third party sourced imagery.

[00078] Figure 6 is schematic diagram of an embedded web link arrangement for access to embodiments of the present invention by means of a digital communications device.

[00079] Figure 7 is a 2D representation of a floorplan showing heat map output superposed on the floorplan in

accordance with a further embodiment of the present invention.

DESCRIPTION OF EMBODIMENTS

First Preferred Embodiment

[00080] Figure 1 is a schematic diagram illustrating a method for creation of a mixed reality 3-D experience from a two-dimensional representation of a three-dimensional

structure when the three-dimensional structure is located in a predetermined environment according to one preferred

embodiment of the present invention.

[00081] Figure 2 is a schematic diagram showing various channels to produce the three-dimensional structure according to any one of the preferred embodiments of the present

invention .

[00082] Further Preferred Embodiment

[00083] With reference to Figures 3A and 3B, there is provided a method for accurately aligning in 3D space a 3D model of a building on a live video view of a site using a 2D siteplan according to one preferred embodiment of the present invention .

[00084] Stage 1: remove the floor plan from the site map to produce an isolated siteplan view containing plan artefacts;

[00085] Stage 2: onsite pan camera to detect onsite

artefacts on the horizontal plan;

[00086] Stage 3: create a combined 2D top down view of detected onsite artefacts;

[00087] Stage 4: Overlay the artefact 2D view on the

isolated siteplan view and virtually shift in order to

correlate plan artefacts with onsite artefacts;

[00088] Stage 5: match at least selected onsite artefacts to plan artefacts thereby to reference 3D model against plan artefacts .

ELECTRONIC CREATION OF SITEPLAN

[00089] In a further preferred embodiment a siteplan may be electronically generated from a floorplan and external image data.

In one form, the external image data, as illustrated in Figure 5 may be derived from satellite imagery obtained from satellites 221 and referenced against GPS or other like spacial coordinates. As shown in Figure 5, a 2D overhead image 210A of a site 210 together with at least one spacial coordinate 211 located within the site 210 is transmitted via spacial imagery server 220 over internet 212 to augmentation server 213. Augmentation server 213 receives a 2D plan 214 from an external source. In one form, the 2D plan 214 may be a 2D plan 214 of a building structure. Augmentation server 213 renders a 3D image of the 2D plan 214 and then locates the 3D image 215 with reference to the spacial coordinates 211. The resulting mixed reality image 216 may then be viewed via an imaging device 217. In one form, the imaging device 217 may be a digital communications device which includes a visual display 218 and radio communications capability 219. In a particular form the imaging device is a mixed reality headset.

Embedded Web link

[00090] Figure 6 is schematic diagram of an embedded web link arrangement for access to embodiments of the present invention by means of a digital communications device.

Superposed heat map

[00091] Figure 7 is a 2D representation of a floorplan showing heat map output superposed on the floorplan in

accordance with a further embodiment of the present invention.

[00092] In a preferred form, the heat map represents virtual person time spent at locations within the mixed reality image.

INDUSTRIAL APPLICABILITY

[00093] The above described methodology can be applied to depict three-dimensional representations of constructs such as dwellings on the actual site for which they are intended. This may be accomplished whilst the dwelling is still in the design phase .