Preciously this invention relates to context association devices for Multimedia using mark-up.

Background of the Invention Reference Cited United States Patent Documents (USPD): 5,404,435 Rosenbaum April 4,1995 5,471,318 Ahuja, etal. November28, 1995 5,742,736 Haddock, April 21, 1998 5,838,313 Hou, et al. November 17,1998 5,943,402 Hamel, et al. August 24,1999 5,915,091 Ludwig, et al. June 22,1999 5,978,835 Ludwig, et al. November 2,1999 The following description specifies present communication methods, technology, devices and systems in the context of background of the invention, identifies certain prior art items, addresses to the various problem, disadvantages and dearths associated with the prior art and offer effective solution to overcome the problem. Accordingly various aspects of the prior art system, technology, devices and the method are enumerated in the following description.

Email: The use of email is prevalent and in a way has become a necessity. Email, short for electronic mail, has become the way of communication whether it is for official or personal use.

Email allows any person with Internet access to send written messages, files and pictures to any other person with Internet access, anywhere in the world. Internet mail delivery works through what is called a client-server system. The email server is used to store email messages from the Internet. The email client software, a program installed in the PC or the web browser itself, is used to retrieve messages from the email server. An individual can reply to an e-mail by either creating a new message or by inserting new information in the existing e-mail.

Voice Mail : A Voice mail systeni enables a person to send and receive voice across the Internet.

Voice mail needs Multimedia PC and voice mail client software to record, playback, and send and receive voice mails. A voice mail server stores messages and makes them available to the user in several ways. The user can access voice mails using a PC, telephone, or a mobile Internet device. Call centers, which are based on voice mail, handle a company's customer service by providing automated call back and answering feature, product inquiries and product/technical support.

Chat : Chat programs enable a person to instantly communicate with people in real-time. Chat application lets a user to communicate to another user connected to the Internet either in the form of text or voice. To use chat, the user has to enter some simple information and then pick a "room"from the many standard and user-created subjects. A"room"contains group of users who discuss about common topics that include news, finance, and entertainment. Chat can also be used to send private messages.

News Groups and Message Boards : A newsgroup is a discussion about a particular subject consisting of notes written to a central Internet site and redistributed through Usenet, a worldwide network of news discussion groups. The collection of posted notes on each subject known as a newsgroup. Most newsgroups are hosted on Internet-connected servers, but they can also be hosted from servers that are not part of the Internet. Message boards, similar to discussion groups, lets a user to express an opinion and read what others have to say on it.

Universal Messaging Services: Universal Messaging Services enable a user to access messages in any format, no matter how those messages were sent. The user can listen to c-mail or fax

messages by phone, or read voice mail messages on computer, hand-hetd persona) digitat assistant or TV. Ail incoming messages reside in single place in an Internet Protocol-based network, in the form in which they were sent. When the user decides to retrieve the messages, they are converted into the form the user wishes at the time-voice or text, and delivered to one place.

In USPD 5,404,435 to Rosenbaum on April 4,1995 PN/5404435 for using keywords to retrieve multi-media data stored in a database. There is no provision for any type of conferencing and do not provide for any real-time communication.

In USPD 5,471,318 to Ahuja, et al. on November 28,1995 PN/5471318 which claims to provide long-term multimedia conference is however limited in the time line and do not offer an user interface through which users can actually manipulate the conferences offline.

In USPD 5,742,736 Haddock, on April 21,1998 describes a product, which is a device for managing voice data automatically linking marked message segments to corresponding applications. Thus, while such system performs similar functions its scope is limited to the extent that it does not allow the user to manipulate any type of multimedia object.

In USPD 5,838,313 to Hou, et al. on November 17,1998 describes a product that is basically a reporting system with recording and playback of dynamic annotation options. It does not give complete control for handling the dynamic annotations to the user and the user cannot create any offline session using the past annotations.

In USPD 5,943,402 to Hamel, et al. on August 24,1999 describes a product that claims to annotate and edit voice messages via acoustic bullet points. Here marking a particular voice segment alone is possible but it does provide for the interfacing of different multimedia objects and does not provide for the users to insert/delete segments of any other multimedia type. It does not have provisions for the newly constructed segment to be stored and sent as a mail to any user. It is primitive in the lack of feature for the segment to be accessed using a PC, wireless device or a telephone.

In USPD 5, 915, 091 to Ludwig, et al. on June 22,1999 describes a product that claims to provide synchronized video conferencing that is interoperable across different computer & network operating system platforms. However the system is limited by its versatility by not providing an interface for a PC, mobile and telephone client. It also does not provide an interface through which users can actually manipulate the conferences offline.

In USPD 5,978,835 to Ludwig, et al. on November 2,1999 talks about transfer of multimedia data over LAN's & WAN's and Real-time Conferencing. Thus, while such systems may bc useful in limited contexts, they do not provide for Markups in Real-time as well as offline multimedia conferencing.

For these existing systems, there is no mechanism to make a visual arrangement for related multimedia data included within a report. It is an object of the present invention, to make the dynamic annotation worthwhile by providing a way for the user to arrange multi-media data in a convenient manner so that related media data can be close together when dynamic annotation is performed. The present communication methods uses technology and system, which allow free flow of communication using voice and text. However, there are certain essential features that these applications do not provide interactive offline communication and certain other features that this product addresses. The issues that are not addressed by present communication methods such as Internet Applications form the objective of the invention.

Objectives of-the Invention It is the primary object of the invention to invent a novel context association application to act as a tool or an application that helps to enable insertion of multimedia objects in an audio, text or video message received through any device like telephone, PC or a mobile telephone/PDA device, record and/or replay or send the modified/grouped message.

It is another object of the invention to invent a novel context association application to link one or more voice message to other voice or text or video stored, saved or recorded messages.

It is another object of the invention to invent a novel context association application to synchronize voice message or segment with text or video message/segment.

It is another object of the invention to invent a novel context association application to use voice commands utilize the novel features.

It is another object of the invention to invent a novel context association application to enable both offline and real-time communication.

Further objects of the invention will be described in the ensuing description.

Summary of Invention Various components, which form part of the invention, are described in the ensuing description.

Markup Markups are digitized representation of the link between 2 multimedia segments. Digitized markups can be used to prepare messages containing multimedia segments for subsequent handling in a business process. Digitized markups provide additional benefits that existing too ! s or applications do not : Digitized markups can be added, moved, and deleted at will without altering the original content.

'The attributes of digitized markups (such as position, color, size, text, and visibility) can be readily modified.

Digitized markups can be linked to multimedia segments, to other files or documents on the user's system, or to Universal Resource Locators (URLs) on the World Wide Web. Such markups become hypermedia jumps, similar to those encountered on the Web.

Groups of digitized markups can be named. The named groups can then be selectively acted upon.

Markups for multimedia segments can be saved separate from the multimedia data. The marks can then be merged with the multimedia data on the fly to create a chain of multimedia segments forming an offline conference consisting of synchronized multimedia segments. Linked markups function aa hypermedia jumps when they are accessed from a device.

Markup functions provide tools for making and manipulating markups. Depending on multimedia segment and the device used to access them, the end user can: 'Create, move, resize, edit, and delete markups.

'Switch between multimedia segments conveniently Modify markup attributes (such as color, size, text and visibility) Group markups and act upon them (such as controlling their visbility) Save markups as part of the multimedia segment Link markups to pages in image documents, to other files on the user's system, or to Universal Resource Locators (URLs) on the World Wide Web. Such markups become hypermedia jumps. When users click on them, the destination media or Web page appears.

Markup groups make it possible to organize and manage digitized markups, giving the end user the capability to: Logically organize markups into functional groups; for example, Positive Review of a book, Negative Review of a book, and so on.

Toggle the visibility of the groups; for example, to Turn Positive Review On a Book, Turn On All Comments, and so on.

A group can contain one or more markups and multiple groups can be associated with a single image/visualization.

Voice Links are markups that function the same as hypertext jumps typically encountered in help systems and on the World Wide Web. They provide a way to navigate within multimedia segments, documents and to Web pages.

When accessed, Voice Links become hypermedia jumps. When users click on them, the destination multimedia segment appears.

Context Associate Application for Multimedia enables a user to insert voice or a multimedia segment when listening to a message. The user is able to insert voice at any desired position of

the message being played, just like typing text between sentences. Context Associate Application for Multimedia then creates a Segment for the inserted voice, and send it to the server, where it is marked and linked to the respective multimedia segment.

Threading When a user sends new message or replies to a message, Context Associate Application for Multimedia creates what is called as"Voice Segments". Voice Segments are digitized stored representation of voice. Context Associate Application for Multimedia maintains threading information for every Voice Segment, by marking them and linking them to other Voice Segments whenever the user wants (see Figure 3). This way a user can post voice messages and have other users leave opinions on the same message in the server.

Insertion Context Associate Application for Multimedia enables a user to insert voice when listening to a message. The user is able to insert voice at any desired position of the voice message being played, just like typing text between sentences. Context Associate Application for Multimedia then creates a Voice Segment for the inserted voice, and sends it to the server, where it is marked and linked to the respective voice segment.

Synchronization Context Associate Application for Multimedia integrates text and voice in one display. by synchronizing the voice segments with the text. For example, a user might want to display relevant text while he/she is speaking on a topic. With Context Associate Application for Multimedia, the user is able to add text to the Voice Segment at the desired time and duration.

This enables users to view messages in presentation style and improve clarity and user interaction.

Filtering Context Associate Application for Multimedia allows filtering of voice segments (with text and graphics). The user can search for specific information based on criteria, select the preferred information from the search result by point and click interface, and listen to it right away. The

figure below (Figure 6) shows collection of voice segments posted by 4 users and voice segments of 2 users selected for listening.

Conversational Style Context Associate Application for Multimedia users are able to access conversational style information at any time, since all voice messages are stored in a central server along with the marked links that point to other voice segments. For example, a user might need important information on a certain topic, and post message that contains series of questions. Any user who accesses the message can listen to the questions and insert answers at the end of each question.

When the voice message is played again, it is conversational style, with set of questions and answers played in the right sequence.

STATEMENT OF INVENTION Context Association Application for multimedia devices using markup comprising a multimedia server being connected to data base through a data access, the said multimedia server having accesses to telephone, mobile, PC through Public switch Telephone network, wireless network and through Public switched network gateway, WAP gateway and Internet respectively such that the system acts as a tool or an application that helps to enable insertion of multimedia objects in an audio, text or video message received through any device like telephone, PC or a mobile telephone/PDA/device, record and/or replay or send the modified/grouped message..

BRIEF DESCRIPTION OF DRAWINGS Now the invention will be described in detail with reference to drawings accompanying the specification. Referring now to the drawings in which like reference numbers represent corresponding parts throughout: FIG. 1 illustrates the currently preferred architecture of Context Associate Application for Multimedia FIG. 2 is a block diagram describing the modular flow within Context Associate Application

FIG. 3 illustrates the significance of digitized markup in accordance with the invention FIG. 4 illustrates the significance of an inserted multimedia object in accordance with the invention FIG. 5 illustrates the significance of the multiple multimedia insertions at different points according to the timeline in accordance to the invention DETAILED DESCRIPTION OF DRAWINGS The preferred architecture of the present invention is discussed in detail below. While specific configurations are discussed, it should be understood that this is done for illustration purposes only. A person skilled in the relevant art will recognize that other components, configurations and embodiments may be used without departing from the spirit and scope of the invention.

FIG. 1 illustrates the currently preferred architecture of Context Association Application for Multimedia along with the different devices through which anyone can get connected to Context Association Devices for Multimedia in accordance with this innovation.

It is discernible that Context Association Application for Multimedia can be contacted through an Internet/Intranet or a wireless domain using a PC, mobile Internet device or a telephone.

Context Association Application for Multimedia enables users to store and retrieve text, image and audio data in the server across the different domains specified above.

In the architectural diagram shown in FIG. 1 illustrates the various current means, method and mode by which one can have access to the Context Association Application. A Telephone client (1) can access the Context Associate Application through a unique telephone number. The telephone client gets connected through a Public Switched Telephone Network Line (4) (PSTN) through which the telephone access is made possible with the Context Association Application.

The PSTN gateway (6) can either be a hardware or software and is directly connected to the Internet. A client with a Mobile Internet device (2) can access the Context Association Application either via telephone or his miniature browser window through the Internet, a

wireless/cellular network (5) is made available through which wireless access is made possible.

The Wireless Access Protocol (WAP) gateway (7) can either be a hardware or software and is also directly connected to the Internet. A PC client (3) with an Internet connection through, which he can get access to Context Associate Application. The PC client can also be an tntranet client considering that the Context Associate Application has been installed in an Intranct setup.

The Internet (8) is the primary medium through which clients get access to Context Associate Devices for Multimedia. The Context Associate Application Server (9), which is a central computer system that provides file distribution to other systems on the Internet or Intrant. A logical data access layer (10) communicates with a central repository of data storage (11). The central repository for data storage (10) is commonly known as the database. The data can be any multimedia object.

FIG. 2 is a block diagram that describes the functional modules within the audio Context Associate Application Server along with the different clients of Context Associate Application.

The database (11) is where all information is stored. The User Manager module (12) within the Context Associate Application Server (9) authenticates clients. and checks their validity. The Message Manager (13) is the module that processes the client messages and makes it intelligent.

The Client Manager (14) is that which interacts with the clients and processes their requests.

There are various user interfaces for clients using different modes to access the Context Associate Application. There is a user interface for the Mobile client (15), one for the PC client being the web browser (16) and an interface for the telephone client (17). The Context Associate Application (18) is the backbone of the entire invention.

FIG. 3 illustrates that digital markups form the crux of the invention. Multimedia Segments (19) are digitized stored representation of multimedia objects. The figure 3 shows a multimedia segment (19) to which digitized markup are applied to. A media link (20) is a point at which another multimedia segment (19) can be linked. The figurer also shows a logical thread (21). Thc invention maintains this threading information for every Multimedia Segment (19), with the help of markups and links them to other Multimedia Segments (19) upon the request of the user. This way a user can post messages and have other users leave opinions on the same message in the server.

FIG. 4 illustrates the significance of an inserted multimedia object in accordance with the invention. The original Multimedia Segment (22) can be viewed or heard depending on its type.

Another Multimedia Segment (23) that is to be inserted into the original Multimedia Segment using a digitized markup (24) is also shown. The digitized markup (24) enables an insertion of a Multimedia Segment. The final Multimedia Segment (25), which is it combination of the other two Multimedia Segments, is shown. In this way a user can insert a multimedia segment at any desired position of any mutlimedia segment being played just like typing text between sentences.

FIG. 5 illustrates the significance of the multiple multimedia insertions at different points according to the timeline in accordance to the invention. Multiple users (26,27, 28,29) can post different Multimedia Segments to a single Multiple Segments (. 19). The time line (30) is also shown along with the filtered segment (31). The invention allows filtering of Multimedia segments. The user can search for specific information based on a criterion, select the preferred information from the search result by point and click interface, and listen to it instantly.

It is to be understood that the object of the description is to explain salient features of the invention. It is to be noted that within the scope of the invention various amendments and modifications are permissible. It is not intended that this invention be limited to the software arrangement, or operational procedures disclosed. This invention includes all of the alterations and variations thereto as encompassed within the scope of the claims as follows :