METHOD OF CREATING A DIGITAL IMAGE RELATING TO INTERACTION BETWEEN USERS OF AN ELECTRONIC COMMUNICATIONS SYSTEM

FIELD OF THE INVENTION

The invention relates to a method of creating a digital image relating to at least a sub-set of a set of interactions between users of an electronic communications system.

The invention also relates to a system for creating a digital image relating to at least a sub-set of a set of interactions between users of an electronic communications system. The invention also relates to a computer programme.

BACKGROUND OF THE INVENTION

KR 2003-0094753 relates to an avatar photographing method and system. An avatar service server comprises a database server, a web server, a chatting server and a file server. The database server provides a member database, an avatar database, an accessory database and a photograph background database. In the photograph background database, the background name and lending price of the background image file, etc., are stored in case a user takes a picture of the avatar. The backgrounds can be categorised by season (summer, autumn, winter), the type of place (sea, sky, school, etc.), the occasion (birthday, Christmas, etc.), etc. A user can take a picture by using a photograph shooting item in a chatroom, one- on-one dialogue, game room, etc. If the user uses a picture-taking ticket, users connected to the web server among users whom one registers as a friend on-line in advance are enumerated in an invitation candidate list. A photograph shooting screen has a chatting box. This has the advantage of allowing chatting between users who together take a photograph. If a host clicks on a background box with a mouse, and it selects the desired background kind, the background picture displayed on a photograph screen is changed to the selected background picture.

A problem of the known system is that it is limited to collaborative generation of a picture, but is not suited to creating a souvenir of any other type of interaction.

SUMMARY OF THE INVENTION

It is an object of the invention to provide a method, system and computer programme of the types mentioned above which are suitable for creating a relatively informative souvenir of any of a wide range of sets of interactions between users of an electronic communications system. This object is achieved by the method according to the invention, which includes: formulating a criterion for obtaining a background image, obtaining an image element for each respective user, each image element including at least part of a digital image associated with the user, and creating the digital image relating to at least a sub-set of the set of interactions as a composite of the background image and the image elements, wherein the criterion for obtaining the background image is derived by a system for analysing a record of information exchanged by at least one of the users and communicated through the electronic communications system in the course of at least some of the interactions.

By creating the digital image relating to the set of interactions as a composite of the background image and the image elements, a relatively concise, but informative representation of a set of interactions is provided. The substance of the interactions is represented by the background image and the participants by the image elements. Because the criterion for obtaining the background image is derived by a system for analysing a record of data exchanged by at least one of the users and communicated through the electronic communications system in the course of the interactions, the method is suitable for implementation in a wide range of systems, as well as being susceptible of automated execution. In an embodiment, at least one of the digital images associated with a user is obtained on the basis of data caused to be provided to the electronic communications system at a user terminal by the user.

An effect is to incorporate elements into the digital image that are more closely associated with the users. The users are identifiable relatively easily from the digital image that is created.

In a variant, at least the part of the digital image associated with the user is provided to the electronic communications system from the user terminal.

An effect is that the elements are not limited in composition to elements predefined in the system creating the digital image and selected by the users. A user inputs his or her own image.

In a further variant, the digital image associated with the user is received by the terminal from a digital imaging device.

An effect is that actual image data is provided to the system creating the digital image relating to the set of interactions. This system is therefore not limited to preloaded image elements. In particular, the digital image associated with the user can be a photograph of the user. In an embodiment of the method of creating the digital image, the image element obtained for at least one of the users comprises a part of the digital image associated with the user obtainable by extraction from the digital image using an image segmentation technique.

An effect is that backgrounds of images associated with users can be excised so that they are not displayed to any significant extent in the digital image relating to the set of interactions. As a consequence, the digital image relating to the set of interactions can relate to more users without the background being obscured.

An embodiment of the method includes analysing at least parts of at least one potential background image for the presence of parts having a high uniformity compared to at least one of a threshold value and a value obtained by analysing the potential background image.

An effect is to be able to ensure that the background part of the created digital image relating to the set of interactions remains informative when the image elements are added to it to compose the final digital image. A variant includes positioning the image elements outside parts having a low uniformity of a potential background image taken as the background image.

An effect is to place the elements such that as much as possible of the informative part of the background image remains visible in the created digital image relating to the set of interactions. A variant of the method includes selecting one of a plurality of analysed potential background images determined to have at least parts with a relatively high uniformity.

Thus, the background image that is used is selected at least partly on the basis of its suitability for conveying information to a percipient when image elements are added to

it. Backgrounds that are relatively uniform are more likely to have replicated information, so that parts of it can be obscured by added image elements.

An embodiment of the method of creating a digital image includes embedding in the digital image relating to at least a sub-set of the set of interactions information for retrieving at least part of the record from a data storage system.

An effect is to add additional information to the digital image that makes it more suitable as a record of set of interactions, without providing it with a significantly larger data size. A user provided with the digital image is able to obtain more information relating to the set of interactions. An embodiment of the method of creating a digital image includes obtaining the information exchanged by at least one of the users as data in text format.

An effect is to obtain the data in a format that is suitable for relatively many content-based analysis methods. This makes it easier to formulate a criterion that would be satisfied by a background image with contents closely related to the subject of the interactions.

In a variant, the criterion includes at least one keyword obtained by applying a summarisation technique to the data exchanged by at least one of the users.

Keywords are suitable for formulating concise search criteria for a relatively wide range of databases. In an embodiment, the criterion includes at least one keyword, and wherein the criterion is submitted to a system for searching a database of metadata linked to images.

An effect is to be able to obtain a relevant background image relatively efficiently, since the search need not involve actual image data analysis. Moreover, it is a relatively precise way of searching for images with relevant content. An embodiment of the method includes ranking digital images satisfying the criterion by analysing at least one of information representative of the contents of the digital images and at least one property of the digital images.

An effect is to be able to resolve situations in which a plurality of available background images satisfy the formulated criterion. Moreover, the criterion need not be refined, nor need an iterative search be carried out on a database to arrive at a single background image.

In an embodiment, wherein the set of interactions comprises a sequence of interactions, the method includes creating a digital image for each of a plurality of sub-

sequences of the sequence of interactions and composing the created digital images into a sequence.

An effect is to create a more informative souvenir of a set of interactions with a time- varying theme. According to another aspect, the system for creating a digital image relating to at least a sub-set of a set of interactions between users of an electronic communications system is configured to: formulate a criterion for obtaining a background image, obtain an image element for each respective user, each image element including at least part of a digital image associated with the user, to create the digital image relating to at least a sub-set of the set of interactions as a composite of the background image and the image elements, and to derive the criterion by analysing a record of information exchanged by at least one of the users and communicated through the electronic communications system in the course of at least some of the interactions.

The system need not be comprised in the electronic communications system.

In an embodiment, the system is configured to carry out a method according to the invention.

According to another aspect of the invention, there is provided a computer programme including a set of instructions capable, when incorporated in a machine-readable medium, of causing a system having information processing capabilities to perform a method according to the invention.

BRIEF DESCRIPTION OF THE DRAWINGS The invention will now be explained in further detail with reference to the accompanying drawings, in which:

Fig. 1 is a schematic diagram of an electronic communications system; Fig. 2 is an example of a composite image relating to a set of interactions between users of the electronic communications system; and Fig. 3 is a flow chart illustrating a method of creating a digital image relating to a set of interactions between users of the electronic communications system.

DETAILED DESCRIPTION

An example of an electronic communications system 1 is illustrated in Fig. 1. A first terminal 2 and a second terminal 3 for effecting electronic communications are connected to a large-area network 4, e.g. the Internet. The first and second terminals 2,3 can be personal computers, game consoles, mobile telephones, personal digital assistants (PDAs), etc. The electronic communications may be synchronous or asynchronous. They may comprise digital or analogue signals and data or speech.

In the following, it will be assumed that digital data is exchanged between users of the first and second terminals 2, 3, and that the users are identifiable in the sense that messages can be addressed to them. The first terminal 2 is shown in more detail for illustrative purposes, and comprises a central processing unit 5, main memory 6 and a mass storage device 7, such as a hard disk drive. It further comprises a user input device 8, such as a microphone and/or keyboard, and is connected to an output device 9, e.g. a display unit and/or speaker system. Data is exchanged with the second terminal 3 using a network interface 10, e.g. a modem or Ethernet adapter.

In one embodiment, users of the terminals 2,3 address data to each other using an instant messaging protocol. Software implementing an Instant Messaging client is stored on the mass storage device 7 and run by the first terminals 2. The second terminal 3 is similarly configured. Instant messages are logged in a local message history, stored on the mass storage device 7 as a record of a session for instance. Typically, messages will be typed in by users, but the same technology can be used to exchange other types of data, e.g. image data or speech data.

In another embodiment, users of the terminals 2,3 communicate through the intermediary of a central server 11. The central server 11 includes a network interface 12, central processing unit 13 and main memory 14, as well as a mass storage device 15. It runs software implementing a server application for communicating with client applications on the first and second terminals 2,3. As an example, the central server 11 can run an Internet Relay Chat (IRC) server application, or an application for hosting an Internet forum, to which users can post messages. In these cases, the data exchanged by one of the users is not addressed to specific ones of the other participants. The central server 11 stores a record of the data exchanged, e.g. in the mass storage device 15.

Depending on the type of communication, any one of the first terminal 2, the central server 11 or a further data processing device (not shown) can generate a souvenir of a session of communication (direct or indirect) between certain identified users of the first and

second terminals 2,3. To this end, use is made of a record of data exchanged by one or all of the certain users and made accessible to them (e.g. e-mails sent to a particular other user, postings made to a forum of which the certain users are also participants, messages transmitted on an IRC channel which the certain other users are also registered to use, data representing actions of a persona in an online massive multi-user game played with the certain other users, communications between avatars representing the users in a "virtual world", etc.).

The souvenir includes at least an image comprising a background image appropriate to the topic of communication between the users and image elements representing the users. In certain embodiments, it includes a sequence of images, e.g. a video sequence, with at least one such a composite image. In certain embodiments, audio information is added to the image or video sequence. The example of generating a set of souvenir images is illustrated in Figs. 2 and 3.

Fig. 2 shows a composite image 16 relating to a set of interactions between two users of an electronic communications system on the topic of a holiday in China. The image 16 is a composite of a background image 17 relevant to the topic of their interactions and first and second image elements 18,19 including at least parts of respective images associated with the first and second users, respectively.

An embodiment of a method of creating an image such as the composite image 16 is illustrated in Fig. 3. In a first step 20, a record of data exchanged by at least one of the users and communicated through the electronic communications system 1 is obtained. The record is generally of a particular session, e.g. the data exchanged during a particular time interval on an IRC channel, or the e-mails sent by one of the certain users to one or more of the other certain users during a particular time interval. In another embodiment, it comprises all sent or received e-mails of a particular user with a particular subject heading and at least one of the other certain users as addressee. In other embodiments, multicast addresses of Internet Protocol (IP) packets are used to identify data forming a record of a set of interactions between the certain users.

Where the electronic communications comprise data communications, e.g. instant messaging, posts to Internet forums or chat rooms, e-mails, the record obtained in this first step 20 is generally already in a character coded format. Where the data exchanged is analogue data, e.g. a voice recording, the data is digitised and speech recognition technology is applied to obtain in text format the information exchanged by the at least one user.

In the illustrated embodiment, the transcript obtained in the first step 20 comprises a sequence of interactions, i.e. a sequence of messages in a certain order. To create a sequence of souvenir images, the sequence of interactions is segmented (step 21) into subsequences, e.g. of an equal number of messages or of equal duration according to time stamps associated with the messages.

Next, the information in the first sub-sequence is analysed to obtain one or more key words (step 22). The analysis can include stop-word removal, text summarisation techniques and/or lexical similarity estimation. Then (step 23) a search criterion is formulated for inclusion in a search query for submission to a search engine, which submission is effected in a next step 24. This search query is submitted to obtain potential background images for the souvenir image relating to the first sub-sequence of messages.

Referring to Fig. 1, a search engine server 25 is connected to the network 4. It has access to metadata annotating digital images stored on first and second web servers 26,27 '. The metadata of the image is indexed and stored in a large database. When a search query is entered, the image search engine looks up the index, and keywords in the query are matched with the stored information. In one embodiment, metadata stored in a file including the digital image data is used, e.g. where the digital images are included in files according to the Exchangeable Image File (EXIF) format or Tagged Image File Format (TIFF). In another embodiment, the text content of HTML or xML files with embedded images stored on the web servers 26,27 is indexed.

In an embodiment, the search engine server 25 orders the results according to relevance to the query.

An overview of techniques to implement the step 22 of keyword extraction is given in Hahn, U., Mani,L, "The challenges of automatic summarization", Computer Q (11), Nov. 2000, pp. 29-36. An alternative implementation of the steps 22-24 leading to the retrieval of candidate background images is given in Joshi, D. et ah, "The story picturing engine: finding elite images to illustrate a story using mutual reinforcement", Proc. 6th ACM SIGMM international workshop on multimedia information retrieval, New York, 2004, pp. 119-126. The data processing system creating the souvenir image obtains (step 28) data representative of at least some of the data of at least a selected number of images identified by the search engine server 25. These images are dubbed potential background images at this stage. In the illustrated embodiment, the data processing system creating the souvenir image also receives at least some of the metadata associated with these images. In an embodiment,

the data processing system creating the souvenir image retrieves thumbnail images (downscaled images) based on the original images. In an embodiment, it retrieves data pertaining to only a pre-determined number of the images identified and ranked by the search engine server 25, this number comprising the highest-ranked ones of the identified images. Next (step 29), a further ranking is carried out by the data processing system creating the souvenir image. The ranking is carried out by analysing at least one of information representative of the contents of the digital images and at least one property of the digital images. The information representative of the contents of the digital images is, for example, the metadata received in the previous step 28. The properties of the digital images relate to the physical properties of the digital images, at least when displayed. Such properties include colour, brightness, contrast, etc.

As an example, at least parts of the potential background images are analysed for the presence of parts having high uniformity in an absolute or relative sense. In an example, the results of this analysis are used to modify or replace the ranking based on contents. In particular, the ranking is established to favour the potential background images having a relatively high uniformity compared to the other potential background images, e.g. the largest area with a level of uniformity higher than a certain threshold value or the largest average uniformity level across the whole image or a measure based on a combination of these two measures. The rationale behind this ranking is that images with a large uniform area are more suited to the addition of image elements without becoming incomprehensible. Ways of measuring uniformity include analysing the spatial properties of the images obtained in the preceding step 28 by determining spatial frequency components (e.g. DCT coefficients, wavelet coefficients), performing edge detection or using object or face detection. In one particular embodiment, a measure of contrast is obtained (low contrast corresponding to high uniformity). One suitable measure of contrast is a local, pixel-by-pixel measure of the spatial complexity of an image. Given a pixel p and an area P surrounding it, the contrast of pixel p is defined as the sum of the absolute differences between the pixel p and the pixels in the surrounding area P. The differences are defined as the linear combination of the differences between the luminance and chrominance components of the pixels. A precise implementation of a method of obtaining such a measure is described in

Ma, Yu-Fei and Zhang, Hong- Jiang, "Contrast-based image attention analysis by using fuzzy growing", Multimedia '03: Proc. 11 ^th ACM Int. Conf. on Multimedia, NY, 2003, pp. 374-381.

The highest-placed potential background image in the ranking established in the preceding step 29 is selected 30 as the background image used to create the souvenir image. At least the results of the image analysis carried out on the selected potential background images are retained, in order to locate the parts of the background image having high and those having low uniformity. Parts having low (absolute or relative to other parts) uniformity (corresponding to high contrast in the specific example given above) are selected (step 31) in order to place image elements for each of the respective users over them.

The image elements for the respective users are obtained (step 32) separately. The image elements include at least part of a digital image associated with one of the users. In an embodiment, the image elements include at least part of a digital image assigned by the central server 11 to the user and communicated to the other users in the course of the interactions between the users. For example, in a massive multi-player game, the image elements might be game characters corresponding to roles assigned to the users. However, these image elements are not very unique. A more remarkable souvenir image is created when the digital image associated with a user is obtained on the basis of data caused to be provided to the central server 11 by the user. An example is where each user composes or selects the components of its own avatar in an on-line environment or massive multip layer game, in which case image data representative of the avatars is obtained in the step 32 of obtaining user images. A yet more remarkable souvenir image results if the user images are provided to the central server 11 from the first and second terminals 2,3, because they were downloaded thereto or created there. In the illustrated embodiment of Fig. 1, a digital imaging device 33 is connected to an interface 34 of the first terminal 2, for capturing a physical image in the form of digital data. The imaging device 33 can be a web-camera, a digital camera or a scanner, e.g. a photo or diorama scanner. The interface can be a USB (Universal Serial Bus), Bluetooth or Firewire interface, for example.

Using the example of digital images captured by the imaging device 33, a next step 35 includes obtaining an image element comprising a part of the digital image associated with the user. This part is obtained by extraction from the digital image using a background/foreground segmentation technique. Thus, the image part showing the user is separated from the background captured by the imaging device 33. The background/foreground segmentation can be carried out at the first terminal 2 or the central server 11. In this manner, an image element is obtained for each of the users to be included in the souvenir image. The image elements 18,19 in the composite image 16 shown in Fig. 2 are obtainable using a foreground/background segmentation technique and separation of the

foreground. As a result, only the actual portraits of the users are overlaid on the background image 17.

The image elements are added (step 36) at positions in the background image outside the parts having a low uniformity (high contrast). Subsequently, the steps leading to the selection of a background image and the step of adding image elements associated with the users are performed again on the basis of the next sub-sequence of the sequence of interactions, in order to obtain further souvenir images.

In one embodiment, the souvenir images are assembled into a slide show. In another embodiment, a video clip is composed. Transition effects between the souvenir images can be added. The souvenir images are provided (step 37) in a file communicated to the first and second terminals 2,3 or made available for download, e.g. at one of the web servers 26,27 '. In an embodiment, the file is in an HTML or xML format, with an embedded link to a location from which an entire transcript of the interactions between the users on which the souvenir images are based can be downloaded.

As a result, the users are provided with a concise and informative summary, with the possibility of obtaining even more information, without such information being transferred to them at first instance. They can infer the topic of the interactions between them, as well as the participants in the communications, merely by inspection of the souvenir image(s).

It should be noted that the embodiments described above illustrate rather than limit the invention, and that those skilled in the art will be able to design many alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses shall not be construed as limiting the claim. Use of the verb "comprise" and its conjugations does not exclude the presence of elements or steps other than those stated in a claim. The article "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The invention may be implemented by means of hardware comprising several distinct elements, and by means of a suitably programmed computer. In the device claim enumerating several means, several of these means may be embodied by one and the same item of hardware. The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be used to advantage.

In an alternative embodiment, the criterion formulated for obtaining the background image is used to search also images exchanged by the users in the course of their

interactions. In another embodiment, a sound track is added to the souvenir image or images. The sound track can be selected using the same keywords as used to select the background image. In another embodiment, the step 30 includes receiving input from the users, e.g. input selecting one of a plurality of potential background images or input data modifying the ranking established in the preceding step 29.

'Computer programme' is to be understood to mean any software product stored on a computer-readable medium, such as an optical disk, downloadable via a network, such as the Internet, or marketable in any other manner.