[0001] METHOD AND APPARATUS FOR VIRTUAL PRESENCE

CROSS REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority on U.S. provisional application no. 60/669,906, filed on April 11 , 2005, which is herein incorporated by reference.

FIELD OF THE INVENTION

[0003] The present invention relates to a method and apparatus for virtual presence. In particular, the present invention relates to a method and apparatus for the interactive immersion of a silhouette image on a display screen of a user interacting therewith.

BACKGROUND OF THE INVENTION

[0004] Virtual reality systems, including some video games and other such virtual environment systems use real-time motion capture devices to translate the presence of a system user, or player, in the virtual environment of the system.

[0005] For instance, in US Patent No. 5,616,078 for a Motion-Controlled User Entertainment System by Oh for Konami Co. Ltd. and issued on April 1 ^st , 1997, a user wears a set of detectable markers to map his/her motion on a virtual character that will exhibit similar motion in a virtual environment projected on a display screen. The markers are captured by an input device and the captured signals thereof are analyzed by a computing device to generate a skeleton used in animating the virtual character. Such a technique

applies a heavy computation load on the system that affects the responsiveness of the character in the virtual environment and/or requires expensive and specialized equipment to be efficient.

[0006] In US Patent No. 6,863,609 for a Method for Controlling Movement of Viewing Point of Simulated Camera in 3D Video Game and 3D Video Game

Machine by Okuda et al. for Konami Co. Ltd. and issued on March 8 ^th , 2005, a video game player' head is detected by an ultrasonic device or a CCD camera that approximates the position of the player's eyes and translates that position to guide the viewing point of the simulated camera in the game. In such a system, the player may only control the viewing point of a character in the game by moving his/her head and is not particularly immersed in the game.

[0007] In International Patent Application No. WO 2004/072909 A1 for a Method and Apparatus for Real Time Motion Capture by Marks for Sony Computer Entertainment Inc. and published on August 26 ^th , 2004 [hereinafter referred to as the Sony application], a more efficient method for controlling a game character in real time is presented. Yet, to represent a human player with plural joints and moving members, a significant computation load is still applied to the system. Furthermore, though real-time motion capture is used to represent the virtual character on a screen, the disclosed method does not explore the use of a virtual camera to enhance the immersion effect.

[0008] In addition, the Sony application discloses the use of a mirror captured image to create a representation of the player in the game. As a result, the player sees a mirror representation of him/herself that feels unnatural and requires significant adaptation from the player since the character on screen and the player share a same left and right side but not a same orientation. Ultimately, the character on the screen is looking at the player, like in a mirror, but not where the player is looking.

[0009] As it is known in the art, the player controlled character in a third- person 3D game (or an avatar in a virtual environment) is more intuitive to the player when the virtual camera used to generate the display is behind the controlled character because the controlled character and the player perception of the display share the same direction, i.e. left is left, right is right, forward is forward and backward is backward, both on screen and using the controls. Yet, since conventional capture systems involve a front facing capture camera, conventional techniques using captured images from these systems can't be implemented without first simulating the skeleton of the controlled character, as with the Antigrav game published by Sony on the EyeToy.

[0010] As it is also known in the art, the lag time due to processing involved in the representation of a motion controlled character (or avatar) has a crucial effect on the quality and effectiveness of the virtual experience. When the lag time increases, the responsiveness of the character decreases and the quality of the virtual experience is significantly reduced.

[0011] As a result, a system is needed to provide greater user immersivity with reduced computation needs in order to achieve a maximum level of responsiveness and detail in the virtual representation of the user in the virtual environment. The present invention, described herein and with reference to the appended illustrative drawings, provides a method and apparatus for interactive presence, and alternative embodiments thereof, that overcome the above and other drawbacks of prior art virtual presence systems.

SUMMARY OF THE INVENTION

[0012] More specifically, in accordance with the present invention, there is provided a virtual presence system for generating on a display a virtual

representation of a user positioned in front of the display, the system comprising an image capture device for capturing an image of the user; and a computing device, the computing device being configured to communicate with the image capture device to generate the virtual representation using the user image and communicate the virtual representation to the display. The virtual representation comprises a mirrored silhouette transformation of the user image such that the virtual representation shares a substantially same orientation as the user.

[0013] Also in accordance with the present invention, there is provided a method for immersing a user positioned in front of a display in a virtual environment rendered on the display, the method comprising the steps of:

acquiring an image of the user;

generating a virtual representation of the user, the virtual representation comprising a mirrored silhouette transformation of the user image; and

displaying the virtual representation in the virtual environment.

[0014] Further in accordance with the present invention, there is provided a computer-readable storage medium having a computer-readable program embodied therein for directing operations of a system comprising an image capture device and a data processor in communication with the image capture device and a display, wherein the computer-readable program comprises instructions for operating the system to immerse a user of the system positioned in front of the display in a virtual environment rendered on the display in accordance with the following:

acquiring an image of the user via the image capture device; generating, using the processor, a virtual representation of the user by applying a mirrored silhouette transformation to the user image; and

displaying the virtual representation in the virtual environment on the display.

[0015] Still further in accordance with the present invention, there is provided a computer-implemented method for generating on a display a virtual representation of a user positioned in front of the display, the method comprising the steps of:

acquiring an image of the user via an image capture device;

automatically generating the virtual representation of the user by implementing a mirrored silhouette transformation of the user image; and

communicating the virtual representation to the display.

BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Having thus generally described the nature of the invention, reference will now be made to the accompanying drawings, showing by way of illustration, illustrative embodiments thereof, and in which:

[0017] Figure 1 is a diagram illustrating the basic configuration of a virtual presence system in accordance with an illustrative embodiment of the present invention;

[0018] Figure 1A is a diagram of an image captured by an image capture device in the system of Figure 1 ;

[0019] Figure 1 B is a representation of a silhouette image extracted and mirrored from the captured image of Figure 1A;

[0020] Figure 1C is a representation of a screen image presented on a display screen in the system of Figure 1 incorporating the silhouette image of Figure 1 B therein;

[0021] Figure 1 D is a representation of a virtual environment corresponding to the screen image of Figure 1C;

[0022] Figure 2A is a diagram illustrating a user of the system of Figure 1 at five successive positions relative to an image capture device thereof;

[0023] Figure 2B is a representation of a virtual environment incorporating successive silhouette models of the user at virtual positions respectively corresponding to the five positions of Figure 2A;

[0024] Figure 2C is a diagram illustrating a captured image of the user at each successive position of Figure 2A and their respective silhouette image.

[0025] Figure 3 is a diagram illustrating a first use of a virtual camera in the system of Figurei to frame a specific region of the successive silhouette images of Figure 2B in the virtual environment of Figure 1C;

[0026] Figure 4 is a diagram illustrating a second use of a virtual camera in the system of Figurei to frame a specific region of the successive silhouette images of Figure 2B in the virtual environment of Figure 1C;

[0027] Figure 5 is a first exemplary screen display of the illustrative embodiment of Figure 1;

[0028] Figure 6 is a second exemplary screen display of the illustrative embodiment of Figure 1 and Figure 7 is a general overview of an exemplary data processing system and method to be implemented by the computing device in accordance with an illustrative embodiment of the present invention

DETAILED DESCRIPTION QF THE ILLUSTRATIVE EMBODIMENTS

[0029] Referring to Figure 1 , a virtual presence system, generally referred to using the reference numeral 10 and in accordance with an illustrative embodiment of the present invention, will now be presented. The virtual presence system 10 generally comprises an image capture device 12 for capturing images of a user, as in reference 14 of Figure 1A, a display 16 comprising a display screen 18, and a computing device 19 for interpreting the captured images 14 provided by the image capture device 12 and communicating a corresponding image, as in reference 20 of Figure 1B ₁ for display on the screen 18. As a result, the processed image 20, derived from the captured images 14 of the user, Is provided on the screen 18 (Figure 1 C) and virtually immersed in the virtual environment 22 of the system 10, as illustrated in Figure 1 D.

[003O] To implement the system, various image capture devices 12 may be considered. For instance, the image capture device may comprise a digital video camera coupled to a screen grabber software or hardware interacting with and/or in the computing device 19, an analog camera providing a continuous video feed to an analog-to-digital (A/D) converter used in conjunction with the computing device 19, as well as any other conventional or specialized imaging device such as infra-red (IR) devices, thermal imaging devices, sonar devices and the IiKe. Other such imaging devices will be apparent to a person of ordinary skill in the art.

[0031] Furthermore, various display devices 16 can also be considered to project a combined image, as in Figure 1C. For instance, the display could comprise an LCD screen, a CRT screen, a plasma screen, or other such conventional or specialized display screens. In addition, one or more image capture devices, as in 12, may be coupled with one or more display devices, as in 16, to enhance the desired visualization effect.

[0032] Also, the computing device 19 may include a personal computer (PC) such as a desktop or laptop PC, a game console, or other such computing and interactive image processing and rendering devices. Ultimately, the image capture device(s) 12 and the display device(s) 16 work cooperatively through the computing device 19, which may implement various image processing and/or combination instructions and algorithms via a single or a series of computer-readable programs stored, maintained, embedded or loaded therein, to provide the desired visual effect of an immersive 3D virtual environment 22. To do so, the computing device 19 may comprise various components such as various static and removable ROMs and other such computer-readable storage media (e.g. CD, DVD, CD-R, CD-RW, DVD-R, DVD-RW, etc.) to store and provide the various software algorithms and instructions used to create, control and display the virtual environment 22 as well as generate, control and display the virtual representation of the user, as in 24, in this virtual environment 22. The computing device 19 may also comprise one or more processors, possibly interacting with active RAM, to implement and control the various software and hardware components, instructions and programs, and other such components that should be readily apparent to a person of skill in the art.

[0033] The computing device 19 may also further comprise device inputs and outputs for connecting thereto, for example, the image capture device 12, the display device 16, various interactive controllers and other such peripherals

optionally required for a given application. Namely, a video game application may require user inputs from a game controller, such as the ubiquitous game pad, or other such controllers including, but not limited to, joysticks, interactive shooting devices, and the like. Other virtual reality applications, such as virtual training and simulation systems (e.g. military training, flight simulations, etc.) may require the use of various types of pointers, a keyboard, a scrolling device or other such user input devices. As will become apparent to a person of skill in the art, various other device configurations may be considered without departing from the general scope and nature of the present disclosure.

[0034] With particular reference now to Figures 1 C and 1 D, the image 20 projected on the screen 18 is illustratively represented by a translucent mirrored silhouette transformation (hereinafter referred to as a silhouette image 20) of the captured image 14. In other words, the captured image 14 is isolated by the computing device 19, inverted from left to right about a vertical axis, and transformed into a translucent silhouette image 20 (Figure 1 B) for display on the screen 18 (Figure 1C).

[0035] Using this approach, the user and the silhouette image 20 share a substantially same orientation. That is, the orientation of the user in the real world is substantially transferred to the silhouette image 20 on the screen 18 (Figure 1 C), and conceptually in the virtual environment 22 (Figure 1 D). Consequently, the right, left, forward and backward directions of the user in the real world (Figure 1) are substantially the same in the virtual environment 22 for the virtual representation 24 of the user therein, hereinafter also illustratively referred to as the silhouette model 24 (in general the terms virtual character, virtual player, virtual model and silhouette model may all be used herein to refer to the virtual representation 24 of the user in the virtual environment 22). Consequently, the manoeuvrability of the silhouette model

24 within the virtual environment 22, whether to move around or to selectively interact with various virtual objects 23 therein, can be more intuitive than with conventional systems. Generally, a greater feeling of immersion is attained increasing user comfort and dexterity with the system 10.

[0036] As will be apparent to a person of skill in the art, various image processing and combination software and hardware algorithms may be implemented by the computing device 19 to produce similar effects. Namely, a user image may first be extracted from the captured image 14 and then transformed into a silhouette representation thereof for subsequent reversal and combination on the screen 18. As will be discussed further hereinbelow, other image rendering and transfer techniques including, but not limited to, image outlines, graphical overlays, shadows, 3D meshes and mappings and the like, may also be used to produce a similar effect. For the sake of simplicity, the following discussion will concentrate on the use of a silhouette image, as in 20 of Figure 1 B, to represent the user via silhouette model 24 in the virtual environment 22, though the above and other user visualization techniques may be considered therein without extending the scope and general nature of the present disclosure.

[0037] To generate the silhouette image 20, various image processing and isolation techniques and methods may be considered. For example, the silhouette image 20 may be obtained from the captured image 14 using a difference matte. This technique requires that a background image be captured without the user. The silhouette image 20 is then computed once the user is in place in front of the matte by detecting color differences of the pixels respectively corresponding to the background image and the captured image 14. A detected color difference greater than a predefined threshold will identify a pixel that is to be part of the silhouette image 20, thereby isolating the image of the user from the background. A person skilled in the art will understand

that any of the other conventional or proprietary difference matte algorithms nnay be considered to achieve a similar result.

[0038] In a further example, the generation of the silhouette image 20 could be obtained from the captured image 14 using chroma keying. This method requires a uniform color (usually a blue screen) to be placed behind the user. Or again, the generation of the silhouette image 20 could be obtained from the captured image 14 using luma keying that is based on the use of specific lighting to isolate the user's image. Various image processing software and/or algorithms are readily available to implement these and other such conventional and specialized methods and should be apparent to a person of ordinary skill in the art. Also, other image identification and isolation methods and techniques, optionally using various other imaging devices such as described hereinabove, can be considered without departing from the general scope and nature of the present disclosure.

[0039] With reference now to Figure 1 D, the silhouette model 24, possibly representing a virtual character in a video game or a video entertainment system, is conceptually defined by the projection 26 of the silhouette image 20 (Figure 1 B) on a virtual panel 27 in the virtual environment 22, the back of the silhouette model 24 referring to the side on which the silhouette image 20 is projected. Figuratively, the projected image 26 is viewed by a virtual camera 28 positioned behind the silhouette model 24 and providing a rear view thereof such that the user and the silhouette model 24 share a substantially same orientation.

[004O] As illustrated by Figures 2A and 2B, the projection 26 is adapted to compensate for the displacement of the user relative to the image capture device 12, which displacement is also used to move the silhouette model 24 in the virtual environment 22. For instance, as the user moves from positions

A to E in the real world (Figure 2A), the position of the silhouette model 24 moves from corresponding positions A' to E' in the virtual environment 22 (Figure 2B), yet the size of the silhouette remains the same. Additionally, as the user moves from side to side, the silhouette model 24 can also move accordingly.

[0041] In Figure 2C, the captured images 14A to 14E and their respective silhouette images 2OA to 2OE are presented for the aforementioned positions A to E in the real world of Figure 2A. As the user moves backward from position A to E, the size of the captured and silhouette images 14 and 20 change within their respective frames 30 and 32. Yet, as illustrated in Figure 2 B, the size of the silhouette model 24 is substantially unchanged. Note however that the shape of the silhouette model 24 may still change if the shape of captured image 14 and silhouette image 20 are changed due to user motion and/or action.

[0042] Consequently, though the actual size of the silhouette image 20 may change, its projection on the silhouette model 24 will not. As will be discussed further hereinbelow, this will allow the position of the silhouette model 24 in the virtual environment 22 to change based on the size and/or position of the captured image 14 and/or the silhouette image 20 within their respective frames 30 and 32 without necessarily altering the size of the silhouette model 24 in the virtual environment 22 and on the display screen 18.

[0043] To position the silhouette model 24 within the virtual environment 22 and to adapt the projection 26 to compensate for the displacement of the user relative to the image capture device 12, an estimate distance of the user from the image capture device 12 is used. Illustratively, by comparing the number of pixels used to render the silhouette image 20 of a selected part or portion of

the user at a given position (portions/parts including, but not limited to the user's head, arm, hand, torso or again any object carried by the user), with the number of pixels used to render a same part or portion at a known position, the distance between the two positions may be estimated.

[0044] Generally, the distance of an object from the image capture device 12 will be inversely proportional to the square root of the number of pixels required to image that object. Consequently, by taking the ratio between the number of pixels needed to render an object at a reference position where the distance between the image capture device 12 and the object is know or estimated and the number of pixels needed to render a same object at an unknown position and, by taking the square root of that ratio and multiplying the resulting value by the known distance, the distance of the unknown position may be estimated. An appropriate scaling factor may also be used to transfer position information to the system 10 and activate a customized displacement or an action of the virtual silhouette 24.

[0045] By implementing such a calculation in the system 10 in real-time for each captured image 12 and corresponding silhouette image 20, or again periodically at a predetermined imaging rate, the position of the user relative to the image capture device 12 may be calculated in real-time. This information can then be used to control the position and orientation of the silhouette model 24 in the virtual environment 22, and that again in real-time.

[0046] As will be apparent to a person of skill in the art, other such methods and algorithms may be used to calculate user distances and motion without departing from the general scope and nature of the present disclosure. For instance, the average number of pixels used in each scan line of the silhouette image 20, or of a predefined part or portion thereof, may be used to linearly interpolate the distance of the user from the image capture device 12.

Alternatively, the distance between the user and the video capture device 12 may be computed using a tracking object having a known shape and size and identifying its relative distance to the image capture device 12, and thus the distance of the user carrying it, using conventional image recognition techniques. In this latter method, tracking object orientation, and thus user orientation, may also be calculated by tracking variations in the captured shape thereof. For example, a circular object may provide an elliptical image when the object is rotated, and thus provide a means to establish the object's orientation based on the eccentricity of the perceived ellipse.

[0047] Additional position tracking systems may also be coupled to the system to provide distance measurements to the system 10. For instance, one or more sonic and/or optical radar devices and techniques may be used in conjunction with the image capture device 12 to provide accurate distance measurements. Such complementary or supplementary positioning device(s) could be coupled to the system 10 to provide user distance and positioning data thereto to supplement data acquired via the image capture device 12. Ultimately, a single, or a combination of distance and positioning devices and techniques may be considered to implement the system 10 without departing from the general scope and nature of the illustrative embodiments.

[0048] Referring back to Figures 2A and 2B, the system 10 may use the estimated distances of the user at each position A to E in the real world (Figure 2A) to position the silhouette model 24 at corresponding positions A' to E' in the virtual environment 22 (Figure 2B). Yet, as illustrated in Figure 2C, as the user moves closer to the image capture device 12, the fraction of the user imaged thereby is reduced such that a representation thereof is limited by the frame 32 of the silhouette image 20. Consequently, a framing 34 of the virtual camera 28 (as illustrated in Figure 1 D) is also constrained by the existing portion of the silhouette model 24, that is the portion rendered by the

silhouette image 20 at any given position. Consequently, if the silhouette model 24 does not include the silhouette of the model's legs, for instance when the user is at a position at which his/her legs are not visible to the image capture device 12, then the framing 34 (Figure 1 D) of the virtual camera 28 should not include the region in the virtual environment 22 where the legs of the silhouette model 24 would be, since visualization data of the silhouette model 24 is not available in this area. (As will be discussed further hereinbelow, estimated and/or extrapolated data could be used to enhance visualization of the silhouette model in areas where real-time data is unavailable.)

[0049] Furthermore, since a mirrored silhouette image 20 is used to immerse the user in the virtual environment 22, the position of the virtual camera 28 may also be constrained to view the silhouette model 24 from behind. Conversely, the orientation of the silhouette model 24 may be constrained to be seen from behind by the virtual camera 28. Yet, various image processing techniques and algorithms could be employed to soften these constraints and provide greater angling capabilities to the system 10.

[0050] Despite these constraints, the present embodiment still provides the ability to implement various camera effects by actively controlling the framing 34 and/or the conceptual position of the virtual camera 28.

[0051] Referring now to Figure 3, the framing 34 of the virtual camera 28 can be restrained to a selected portion of the user's silhouette image 20. Consequently, as the user moves from position A to E in the real world (Figure 2A), the virtual camera 28 maintains a constant framing 34. As will be readily understood by a person of skill in the art, this will provide the visual effect that the silhouette model 24 moves backward in the virtual environment 22 while a constant representation thereof, limited herein by the virtual

camera framing 34, is maintained at every position.

[0052] For this effect, the virtual camera framing 34 may, for instance, be selected to encapsulate the portion of the silhouette image 20 available when the user's position relative to the image capture device 12 represents the closest position interpretable by the processing device 16. That is, for a given application, a range of distances will be interpretable and transposable to the virtual environment 22. A closest distance in that range will thus produce the largest silhouette image 20 transferable and thus the most limiting silhouette image framing 32. By limiting the virtual camera framing 34 to this same framing 32, or to a smaller section thereof, a constant visual representation of the silhouette model 24 will be possible for every user position supported by the application.

[0053] Alternatively, the user position range may be delimited by the selected virtual camera framing 34. That is, the sensitivity of the system to user motion may be calibrated by selecting a minimal virtual camera framing 34 defined to provide an adequate visualization of the silhouette model 24.

[0054] In another alternative, the frame 34 may be dynamically centered on the silhouette image 20, or a pre-selected part thereof, using a pixel centroid thereof. In other words, the system 10 could calculate a conceptual 'center of gravity' of the imaged object by evaluating the position of each imaged pixel in x and y and dynamically center the frame 34 thereon.

[0055] Again, a person of skill in the art will understand that other such frame calibration and alignment methods can be used to produce a similar effect.

[0056] Referring now to Figure 4, the framing 34 of the virtual camera 28 may also be altered dynamically to provide a zooming action. For instance, the

user may use a backward motion to implement a zoom-out function in the application. As the distance of the user from the image capture device 12 increases, the virtual camera framing 34 may also increase, thereby providing a zoom-out effect that may optionally be dynamically coupled to a rearward motion of the silhouette model 24 in the virtual environment 22.

[0057] As mentioned hereinabove, the lateral motion of the user may also be used to control actions of the silhouette model 24 and the virtual camera 28 in the virtual environment 22. For instance, when the user moves to the left, the silhouette image 20 will be moved to the left side of the frame 32 thereof. By detecting this lateral displacement, the silhouette model 24 could also be correspondingly displaced in the virtual environment 22. Alternatively, a position or viewpoint of the virtual camera 28 could be moved to reflect a change in the viewing direction of the virtual silhouette 24.

[0058] As will be apparent to a person of skill in the art, various visualization control algorithms may be implemented to produce such lateral displacement effects. For example, the positioning of the silhouette model 24 could be determined as a function of a static virtual panel, as in reference 27 of Figure 1 D on which the silhouette image 20 is projected to construct the silhouette model 24, that moves forward and back based on the distance of the user from the image capture device 12. The lateral position of the virtual silhouette 24 could then be modified dynamically as the position of the silhouette image 20 moves laterally within the framing 32 thereof.

[0059] Alternatively, a dynamic virtual panel could follow the movement of the virtual silhouette 24. That is, based on the lateral position of the silhouette image 20 within its framing 32, a panel or model cutout could be simulated in the virtual environment 22 to reflect motion of the user in the real world. Though conceptually similar and providing a substantially same result, the

implementation of these two algorithms may vary, each providing their own advantages and/or features.

[0060] Ultimately, the above and various other 3D positioning and visualization algorithm may be implemented to produce the desired effect, that is to control the actions of the silhouette model 24 through the motion of the user.

[0061] As such, a person skilled in the art will recognize that the typical camera movements used in 3D software, like zooming and panning can be available with this system 10, as well as other movements such as moving and dollying. To enhance the visualization of the latter two, additional software and hardware algorithms may be used to address various reorientations of the silhouette model 24 and various perspective inconsistencies attributed thereto.

[0062] Furthermore, other visual effects may be generated by the motion of the user relative to the image capture device 12. For example, the displacement of the user and/or his position relative to the video capture device 12 may be used in conjunction with state information in the virtual environment 22 to control the position and/or the orientation of the silhouette model 24. For instance, in a game setting, a user moving from the left to the right of the image capture device 12 could instruct a silhouette model 24 to orbit around an object rather than to strictly move from left to right. Alternatively, a user moving three (3) feet to the left could instruct a silhouette model 24 to move four times as much to give the user a feeling of power or speed.

[0063] Many other examples may also be considered. For instance, as the system 10 detects a motion of the user, the computing device 19 may be

configured to implement various motions and/or displacements of the silhouette model 24 in the virtual environment 22, or again implement various modifications to a visualization thereof from the perspective of the virtual camera 28. For example, user arm, leg, hand, head and body motions may be mapped by corresponding motions of the virtual model 24 in the virtual environment 22. Also, user motions may be used to implement virtual displacements of the virtual model 24 such as a forward displacement, a backward displacement, an orbital displacement, a rotational displacement and a side displacement. Alternatively, user motions may be detected and used to control various virtual camera perspective modifications, which may include moving, rotating, panning, zooming, orbiting and/or dollying the virtual camera relative to the silhouette model 24 and/or the virtual environment. A person skilled in the art, namely in the art of video game and virtual reality system development, will understand that such controller mapping features and options used to generate and enhance user immersion, impressions, and experiences may be readily adapted to the system 10 without departing from the general scope and nature of the present disclosure.

[0064] In another example, one or more markers may be worn by the user to implement various visual features and options in a given application. For instance, various conventional algorithms may be used to compute the location of these markers from the image capture device 12 and use these locations to allow the user to further interact with the virtual environment 22, for instance for implementing collision detection algorithms with virtual objects, as in references 23 in Figure 1 D.

[0065] Furthermore, conventional or proprietary tracking algorithms may be used to compute the locations of specific user features, for instance the user's face, eyes and/or hands from the image capture device 12 (e.g. tracking on a shape, size and/or color thereof). These locations could then be used to

enhance user interactions in the virtual environment 22, for instance again for implementing collision detection algorithms with virtual objects. In this configuration, the user's specific features (e.g. user's face, eyes, hands, etc.) could be used to act as markers, allowing the system 10 to track these features using conventional and/or modified tracking techniques and/or algorithms. This type of marker detection and tracking could also be used to implement various other actions in the virtual environment (e.g. virtual representation movement, virtual camera movement, etc.)

[0066] Also, in yet another example, the user may carry one or more objects that have distinctive features, like for instance a bright unusual color or a distinctive shape, to be used interactively in the virtual environment 22. For instance, conventional or proprietary motion capture algorithms could be used to compute the location and/or orientation of theses objects from the image capture device 12 to enhance user interactions in the virtual environment 22, such as for collision detection with virtual objects, for interactively displaying the captured objects, or for substituting these objects with virtual objects.

[0067] Additionally, the presentation of the silhouette model 24 may be modified or altered using conventional or proprietary visual effects to reflect user interactions in the virtual environment 22. These effects could include any type of visual add-ons and features, image distortions and enhancements, motion-activated visual cues and actions and the like.

[0068] Also, conventional gesture recognition algorithms and techniques may be used on the data provided by the video capture device 12 to implement user interactions with virtual objects 23 in the virtual environment 22. For instance, the silhouette image 20 could be directly used to compute 2D collisions with virtual objects whose positions are mapped in 2D to a plane, such as virtual plane 27 of Figure 1 D, determined by the position and

orientation of the silhouette model 24. Collision information could then be used to implement various user interactions in the virtual environment 22. Detection of a velocity or direction of a user's motion could also be tracked to further govern the implementation of various user interactions with virtual objects 23.

[0069] For example, the computing device 19 could be configured to implement various predetermined collision actions in the virtual environment based on collision data calculated from a collision between the silhouette model 24 and a given virtual object 23 in the virtual environment 22. Such collisions could be generated from a user-controlled motion of the silhouette model 24 and/or a motion of the given object 23. Possible collision actions could include, but are not limited to, selecting, modifying, moving, removing and/or modifying the given object, or again moving or modifying the silhouette model 24, or a representation thereof, as a result of such collisions. Furthermore, the computing device 19 could be configured to detect and determine whether such a collision was initiated by a user motion or by a preprogrammed motion of a moving virtual object 23.

[007O] As can be seen, the system 10 provides a visually interesting virtual experience that, illustratively using virtual camera 28, augments the immersion sensation of the user in the virtual environment 22 with the added benefit of further deforming and displacing the projection of the silhouette image 20 on the silhouette model 24, which is ultimately seen on the display screen 18. Consequently, the system 10 contributes in hiding the origin of the silhouette image 20 and makes it easier for the user to accept it as an integral part of the virtual environment 22, rather than to perceive it as his own mirrored image.

[0071] The system 10 thus provides the user with the illusion that his/her silhouette has been captured from behind and projected inside the virtual environment 22 and, through this silhouette image 20, he/she can interact in the virtual environment 22 in a natural way. Namely, since the user and the user's visual representation, illustratively provided by the user's silhouette model 24, share a same general orientation, the user is provided with a better sense of direction and can thus better interact within the virtual environment 22. This provides a very intuitive control of the virtual silhouette 24 as the user is provided with the illusion that his/her shadow is cast on the display screen 18 and can directly interact in the virtual environment 22.

[0072] In addition, the system 10 provides the user silhouette, as it would be observed from behind, while capturing the image for the generation thereof from the front of the user. This permits the system 10 to be installed conventionally, such that the user faces both the display 16 and the video capture device 12. Moreover, by positioning the image capture device 12 in front of the user, a frontal image of the user is obtained that provides greater visual cues usable in the implementation of the system 10, such as the position of the hands of the user and any object carried thereby. As will be apparent to a person of ordinary skill in the art, various visual cues such as tags, interactive pointers and the like may be used to enhance the visualization effect and facilitate interaction within the virtual environment 22.

[0073] Also, the use of a silhouette image 20 provides for the easy harmonization of the projected image 26 in a stylized virtual environment 22. For instance, it would be harder to respect the artistic direction of the virtual environment 22 if an actual image was used to overlay the virtual environment graphics. Furthermore, while true image overlays may be intimidating for some users, the use of silhouettes, while still providing an adequate identification of the user on the screen 18, provides a less threatening

experience accessible to all.

[0074] As stated hereinabove and as will be apparent to a person of ordinary skill in the art, other user visualizations may be implemented in the system 10 without departing from the general scope and nature of the present disclosure. For instance, a user outline may be used instead of a silhouette. Alternatively, the user silhouette could be replaced by a graphical representation of a character in the virtual environment 22, such as the hero in a simulated game setting. Furthermore, additional objects and clothing may be added to the user representation, such as a hat, a gun, or other such accessories used to identify the character and various activated features and options in the virtual environment 22.

[0075] Other user visualizations, such as 3D mesh mappings, scaled graphical representations, image and graphical overlays and the like, ranging from the fully opaque to various degrees of transparency to allow better visualization of the virtual objects near the silhouette model 24, may also be considered. Furthermore, silhouette image enhancements including, but not limited to, image replacements and fillers to compensate for missing, corrupt or unavailable image data, image overlays providing uniform or patterned templates for user representation and identification, and other such features may also be considered.

[0076] Ultimately, the system 10 allows to generate a good, complex and responsive representation of the user in the virtual environment 22, that is the silhouette model 24, at low computational cost. Namely, since the silhouette model 24 can be generated partly using image-processing techniques and avoids the complex analysis of the captured image 14 to infer the posture of the user, typically needed with conventional virtual presence models, computational load is significantly reduced and system responsiveness

increased. Furthermore, since calculation of the user's motion about the image capture device 12 is mainly implemented using pixel counting and interpretation techniques, though other techniques may be used cooperatively therewith or instead thereof, computational load is kept low.

[0077] Also, not only is computational load reduced by the above techniques, the position and posture of the user's virtual representation 24 is rendered with a higher precision than generally expected from conventional position and posture calculation techniques. By using a transformation of the user's image rather than a computed graphical representation thereof coarsely derived from positional and postural data, the posture and positioning of the virtual representation 24 is provided with enhanced detail.

[0078] As will be apparent to a person of skill in the art, other embodiments may include multi-player applications comprising plural silhouette models 24 interacting in a same virtual environment 22, either locally or on a shared network of computing devices and displays. For instance, plural users could be imaged by a single image capture device 12 and a user identification algorithm could be implemented to identify each user for immersion into the virtual environment 22. Alternatively, plural image capture devices 12 connected to one or more processing device 19 could extract individual user silhouette models 24 and combine them in a given virtual environment on one or more displays 16. Or again, a virtual environment 22 could be shared between two or more networked systems 10, wherein the silhouette model 24 of a first user of a first system 10 would appear as a virtual object in a second system for interaction with a second silhouette model of a second user, and vice versa.

[0079] Referring now to Figure 5, an exemplary screen display of the disclosed system 10 is presented. In this screen capture 50, a user interacts

in a virtual environment 22 comprising a boxing arena and comprising interactive virtual objects 23 such as an opposing boxer 52. As can be seen, the silhouette model 24 controlled by the user can move within the virtual environment 22 and interact, such as punch, jab and push the virtual boxer 52. Also, the system 10 may be configured such that the opposing character 52 can punch, jab and push the silhouette model 24, thereby inducing modifications thereto (e.g. collision highlights, character reactions, etc.), as it would a conventional virtual character controlled by user in a conventional video game environment.

[0080] In this particular example, the system 10 is calibrated to provide an interactive boxing match between the user and the virtual boxer 52. To enhance the virtual boxing experience, the system 10 will not only track the position of the user, in this scenario by tracking the user's head, but also track the position of the user's hands. For instance, the user could carry colored objects in each hand, namely a bright pink object in a right hand and a bright green object in the left hand for the independent identification and tracking thereof using conventional or proprietary color tracking algorithms (colors should be sufficiently different from other colors in the camera frame to optimize identification and tracking). Alternatively, the skin color of the user's hands may suffice to provide sufficient color distinction to identify the hands of the user. The head of the user could then be independently identified and tracked by using the above-described pixel counting and interpretation techniques with the remaining pixels in the image.

[0081] To calibrate the system, that is to provide a known or estimated distance from which other user distances may be calculated, a calibration stage could, for example, ask the user to align his/her head with an on-screen marker predetermined to position an average user at a desired distance. For instance, the image of the head of an average user could coincide with the

marker when the user stands at approximately four (4) feet from the image capture device 12. To calibrate the hands of the user, the user could be asked to put his/her hands substantially vertically above his/her head, or again lined- up with additional screen markers, such that the hands and head of the user are at a substantially same distance from the image capture device 12 during calibration. A predefined line at the top of the screen could guide this action such that the hands of an average user would exceed that line when standing at the pre-selected distance with his/her hands in the air. In one example, the game could be started once the image of the user's hands has been properly positioned above this calibration line.

[0082] Consequently, the illustrated game setting would comprise a three- body silhouette model defined by the user's head/body at a first position and distance, the user's right hand at a second position and distance, and the user's left hand at a third position. As a result, various options and features could be applied independently to each tracked body and coincide with the motion thereof in the virtual environment 22.

[0083] Yet, though the player interacts with the game in a 3D environment, the projected image 26 remains as a 2D representation of the user. As seen in Figure 6, which provides an exemplary screen display of the same setting as Figure 5 but from the vantage point of an additional virtual camera, the virtual camera projecting the silhouette image 20 remains behind the virtual user and need only project a 2D image thereof. When the player and the silhouette model 24 share a same orientation, the illusion of 3D virtual presence is obtained nonetheless, avoiding the heavy implementation of a full 3D virtual image.

[0084] As will be apparent to a person of ordinary skill in the art, various image enhancing and processing algorithms may be used to improve the

visualization of the system 10, namely to adjust any distortion or perceptual inconsistencies associated with a plural-bodied silhouette model 24. For instance, various simulated image corrections could be implemented to address size inconsistencies between objects tracked at different relative distances from the image capture device 12. For example, in the boxing setting presented hereinabove, the images could be processed to compensate for potential perspective errors induced when the user moves his/her fists towards the video capture device. Without visual compensation, the imaged fist would become bigger when it should be getting smaller since the fist moves away from the virtual camera 28. Using visual compensation for movements of the fists relative to the body, these inconsistencies could be addressed. Yet, even without using any elaborate image processing and compensation algorithms, the system 10 will still provide a realistic immersive virtual experience for the user.

[0085] Referring now to Figure 7, a general overview of an exemplary data processing system and method to be implemented by the computing device 19, in accordance with the above illustrative embodiments, will now be presented. In general, as briefly described hereinabove, the computing device 19 comprises an image input 60 for communicating with the image capture device 12 and an image output 62 for communicating with the display 16. Optional user inputs, as in 63, may also be included to provide additional user interaction with the system 10.

[0086] Also, the computing device illustratively comprises an embedded and/or removable storage memory 64 (e.g. ROM, CD, DVD, CD-R, CD-RW, DVD-R, DVD-RW, etc.) for storing various processing software and/or algorithms, which may comprise a number of instructions for implementing various image processing, integration and immersion algorithms, as well as various data structures for rendering the virtual environment and interactions

generated therein. These exemplary components may include, but are not limited to, virtual environment data, software and/or application program(s), image processing, isolation and rendering software, algorithms and/or application program(s), and virtual world image integration and immersion software, algorithms and/or application program(s).

[0087] Still referring to Figure 7, the computing device 19 also comprises a processor (e.g. CPU) and optional operating system 66 for implementing, illustratively via RAM 68, the various steps and instructions provided by the various components/program(s) stored in memory 64.

[0088] In particular, Figure 7 illustrates an exemplary succession of steps and instructions implemented by the CPU 66. In a first step, the CPU 66 acquires a captured image via the image input 60. The captured image is then isolated and inverted (e.g. background removal, silhouetting, scaling, framing, mirrored transformation). When positioning-dependent instructions and/or effects are available (virtual positioning, visualization, etc.), the CPU 66 may detect a position of the user (e.g. pixel count, marker detection, etc.) and implement these position-dependent instructions/effects. The processed image is then integrated into the virtual environment (e.g. integration of processed image with virtual environment data) and the integrated image communicated to the display 16 via the image output 62. Other optional steps may include generating various visual enhancements (e.g. image transparency, accessories, distortions, character overlays, add-ons, etc.) as well as computing various collisions between the virtual representation of the user and virtual objects in the virtual environment.

[0089] The person of skill in the art will understand that various other steps and instructions may be considered to implement the above system without departing from the general scope and nature of the present disclosure. Also, it

will be understood that the above exemplary steps and instructions may be performed in various orders without significantly altering the end result. Namely, the various image processing and integration steps may be performed in a number of sequences to obtain a like effect.

[0090] It is to be understood that the invention is not limited in its application to the details of construction and parts illustrated in the accompanying drawings and described hereinabove. The invention is capable of other embodiments and of being practised in various ways. It is also to be understood that the phraseology or terminology used herein is for the purpose of description and not limitation. Hence, although the present invention has been described hereinabove by way of illustrative embodiments thereof, it can be modified, without departing from the spirit, scope and nature of the subject invention as defined in the appended claims.