CROSS REFERENCE TO OTHER APPLICATIONS

[0001] This application claims priority to U.S. Provisional Patent Application No.

62/579,790 entitled CLOSING THE LOCALIZATION LOOP IN AR FOR FLOORPLANS filed October 31, 2017 which is incorporated herein by reference for all purposes.

BACKGROUND OF THE INVENTION

[0002] Existing augmented reality (AR) applications in which renderings are overlaid on real time camera views typically suffer from alignment issues due to errors in estimates of camera pose in unknown three-dimensional spaces. Thus, techniques that result in more accurate alignment are needed and disclosed herein.

BRIEF DESCRIPTION OF THE DRAWINGS

[0003] Various embodiments of the invention are disclosed in the following detailed description and the accompanying drawings.

[0004] Figure 1A is a high level flow chart illustrating an embodiment of a process for generating an estimate of a floorplan of a three-dimensional space.

[0005] Figure IB illustrates an example of an estimate of a floorplan.

[0006] Figure 1C illustrates an example of a composite image having alignment errors.

[0007] Figure 2A illustrates an example of a revised or corrected floorplan.

[0008] Figure 2B illustrates an example of an aligned composite image.

[0009] Figure 3 is a high level flow chart illustrating an embodiment of a process for facilitating generation of a substantially perfectly aligned composite view comprising combined rendered and actual camera views.

DETAILED DESCRIPTION [0010] The invention can be implemented in numerous ways, including as a process; an apparatus; a system; a composition of matter; a computer program product embodied on a computer readable storage medium; and/or a processor, such as a processor configured to execute instructions stored on and/or provided by a memory coupled to the processor. In this specification, these implementations, or any other form that the invention may take, may be referred to as techniques. In general, the order of the steps of disclosed processes may be altered within the scope of the invention. Unless stated otherwise, a component such as a processor or a memory described as being configured to perform a task may be implemented as a general component that is temporarily configured to perform the task at a given time or a specific component that is manufactured to perform the task As used herein, the term 'processor' refers to one or more devices, circuits, and/or processing cores configured to process data, such as computer program instructions.

[0011] A detailed description of one or more embodiments of the invention is provided below along with accompanying figures that illustrate the principles of the invention. The invention is described in connection with such embodiments, but the invention is not limited to any embodiment. The scope of the invention is limited only by the claims, and the invention encompasses numerous alternatives, modifications, and equivalents. Numerous specific details are set forth in the following description in order to provide a thorough understanding of the invention. These details are provided for the purpose of example, and the invention may be practiced according to the claims without some or all of these specific details. For the purpose of clarity, technical material that is known in the technical fields related to the invention has not been described in detail so that the invention is not unnecessarily obscured.

[0012] In augmented reality (AR) applications, a rendering is displayed on top of an actual camera view of a physical environment to provide a blended or mixed reality experience. For example, a rendering may be overlaid or superimposed on a real time camera view to generate a composite view that is a different or more enhanced version of an actual view or scene seen by the camera. In such applications, correct or proper placement of rendered objects in camera views relies on accurate estimates of camera pose in three-dimensional space.

[0013] Mobile devices that support augmented reality applications, however, have not yet evolved in terms of sensors and processing power to provide sufficiently accurate estimates of camera pose, especially in real time, resulting in less than ideal AR experiences in many cases. Specifically, drift in mobile device sensors (e.g., over time, with movement, across large spaces) produces cumulative error or noise in camera pose estimates, which surfaces as inconsistencies, mismatches, and/or misalignments in generated composite views. [0014] Techniques for correcting errors in camera pose estimates based on foreknowledge of the physical environment are disclosed herein. The disclosed techniques facilitate substantially accurate alignment in composite views and, thus, result in well registered augmented reality experiences.

[0015] The disclosed techniques may be employed with respect to an augmented reality application that, in some embodiments, comprises client-side and server-side components. In some such cases, for example, an actual view comprising a current field of view of a camera of a client device, e.g., in a viewfinder or display associated with the camera, is combined in real time with a rendering provided by a server to generate a composite augmented reality view at the client device, in some cases based on input or interactions from a user of the device. A client-side device may comprise any computing device that is capable of communicating with an associated server, such as a cellular or mobile phone, tablet computer, laptop computer, virtual or augmented reality device or headset, etc. The disclosed techniques, however, are not limited to client-server architectures and generally may be employed with respect to any other appropriate augmented reality application architectures.

[0016] In some embodiments, a priori information about the physical environment includes foreknowledge that the physical environment comprises a floorplan. That is, it is known that the physical environment comprises an enclosed, indoors space (e.g., a building or part thereof such as a room) comprising elements including a floor, walls, and corners. Given foreknowledge that a physical environment comprises a floorplan, an estimate of the floorplan of the physical environment is first generated. An example of a technique for estimating a floorplan is next described with respect to Figure 1 A. However, generally, any one or more appropriate techniques may be employed to generate a floorplan estimate of a three-dimensional space, e.g., based on views of the three-dimensional space from a camera of a device located in the three-dimensional space.

[0017] Figure 1A is a high level flow chart illustrating an embodiment of a process 100 for generating an estimate of a floorplan of a three-dimensional space. At step 102, a floor plane of the floorplan is identified, e.g., using a standard technique such as the methods used by Apple ARkit API or Google Tango API. At step 104, user input comprising a line specifying where a wall meets the floor is received. For example, a user may draw a line on a display of a user device showing a camera view of the three-dimensional space and align the line drawn on the display with where the wall meets the floor in the camera view to generate the user input of step 104. At step 106, camera pose is estimated. Any one or more appropriate techniques may be employed to estimate camera pose at step 106. For example, camera pose may be estimated based on inertial measurements, simultaneous localization and mapping (SLAM), etc. At step 108, the line of step 104 is placed in three-dimensional space at the location of the wall and aligned with the projection of the line on the camera. Step 110 facilitates iteration of steps 104-110 of process 100 for each wall until all walls have been specified by lines. At step 112, an estimate of the floorplan is generated by extending and joining the lines in three-dimensional space that result from step 108.

[0018] Figure IB illustrates an example of an estimate of a floorplan generated using process 100 of Figure 1A. In Figure IB, the dashed lines are the lines of step 108, the solid lines comprise extensions of the dashed lines, and the intersections of the extended solid lines (cross marks in the figure) comprise corners of the floorplan. As can be seen in Figure IB, distortions exist in the ground plane and walls of the estimate of the floorplan. Errors in the estimate of the floorplan result from errors in the estimates of camera pose at step 106. Figure 1C illustrates an example of a composite image having alignment errors. In this example, errors in the corresponding floorplan estimate (or equivalently camera pose) result in misalignment between rendered objects (the home furnishings items in the image) and real world features (the floor, walls, and ceiling of the room). The white lines are specifically included in Figure 1C to illustrate that errors in the floorplan estimate result in mismatched corners and lines, i.e., the walls and ceiling from the floorplan estimate are misaligned with respect to their real world counterparts in the actual camera view of the room.

[0019] In some embodiments, errors in floorplan estimates may be corrected based on a priori information about floorplans, such as known information about typical (room) shapes, layouts, and dimensions. In some such cases, a cost or error function may be minimized with respect to an estimate of the floorplan to generate a corrected or revised version of the floorplan that more accurately represents actual pose. Such a cost function may include a set of one or more constraints based on foreknowledge of standard geometries (e.g., angles, heights, etc.) and/or other properties associated with floorplans. An example of a constraint that may be associated with the cost function includes constraining walls to meet at right angles (i.e., angles of ±90°). Such a constraint does not result in the snapping of all walls to right angles (which would create further errors) but rather weights walls that are close to right angles more favorably in the cost function, resulting in walls that should be at right angles to actually be at right angles in the revised version of the floorplan and walls that are actually at other angles (e.g., slanted) to remain that way. In some cases, the floorplan itself is not unknown but rather a precise specification of the floorplan in which the device (camera) is located exists and is known in complete detail. In such cases, the constraints comprise the known specification of the floorplan, and the revised floorplan is constrained to match the known specification.

[0020] In various embodiments, any appropriate cost or error functions may be employed.

Such cost or error functions, for instance, may depend on variances from standard values expected with respect to floorplans. An example of a cost or error function having the constraint that walls meet at right angles follows as Equation 1.

[0021] Φ =∑" ₌₁ (90 - ¾) (Equation 1)

[0022] In Equation 1, JVis the number of walls, and 9j are the angles between the jt and

(/+l)th wall. 9j is further specified by Equation 2.

[0023] 0j = f(Cj) (Equation 2)

[0024] In Equation 2, C _j -are the extrinsic camera matrices for cameras used to mark lines on the camera display that align to walls. Cj is further specified by Equation 3.

[0025] (Equation 3)

[0026] In Equation 3, Rj are rotation matrices, and tj are translation vectors.

[0027] Figure 2A illustrates an example of a revised or corrected floorplan generated by minimizing the cost function of Equation 1 with respect to the floorplan estimate of Figure IB. As depicted, the corrected floorplan comprises more accurate angles between walls. Figure 2B illustrates an example of a composite image resulting from overlaying a rendering having a camera pose corresponding to the corrected floorplan of Figure 2 A with an actual camera view. As depicted, the composite image comprises well aligned corners and lines as highlighted by the white lines in the image from the corrected floorplan.

[0028] Figure 3 is a high level flow chart illustrating an embodiment of a process 300 for facilitating generation of a substantially perfectly aligned composite view comprising combined rendered and actual camera views. For example, process 300 may be employed with respect to an augmented reality application. At step 302, an estimate of a floorplan of a three-dimensional space is generated based on one or more views of the three-dimensional space from a camera of a device located in the three-dimensional space. For example, the estimate of the floorplan may be generated using process 100 of Figure 1 A. At step 304, a cost function is minimized with respect to the estimate of the floorplan generated at step 302 to generate a corrected or revised floorplan having less error than the estimated floorplan. For example, the cost function may comprise the cost function of Equation 1. At step 306, a pose of the camera corresponding to the corrected floorplan is determined. At step 308, a rendering having the determined camera pose is displayed in an actual field of view of the camera, resulting in a composite, augmented reality view.

[0029] As described, the disclosed techniques constrain three-dimensional spaces to spaces comprising floorplans and facilitate the determination of more accurate estimates of camera pose, which is especially useful in real time augmented reality applications. The disclosed techniques are not limited to augmented reality applications but generally may be employed with respect to a variety of other applications. For example, substantially perfect alignment allows the use of camera images for extracting consistent textures for three-dimensional model rendering.

[0030] Although the foregoing embodiments have been described in some detail for purposes of clarity of understanding, the invention is not limited to the details provided. There are many alternative ways of implementing the invention. The disclosed embodiments are illustrative and not restrictive.