CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims priority to U.S. Provisional Patent Application

No. 62/142,710 filed April 3, 2015, the contents of all of which are expressly incorporated by reference herein.

TECHNICAL FIELD

[0002] Teachings relate to consuming and enriching electronic content on web enabled devices and sharing it over the Internet, and more specifically, to consuming web pages, PDFs and eBooks (ePubs), videos and audio recordings, on PCs and mobile devices, enriching such content with special annotations and sharing such enriched content or annotations via the Internet and social media channels. BACKGROUND

[0003] When electronic content of any form is consumed by a person interested in gathering information, only a fraction of that content is really relevant. Therefore, consuming content can be very ineffective and tiring because people have to read/browse/ scan through irrelevant passages that cany little or no relevant information.

[0004] Accordingly, there is a need for a system and method for tools and services to enable consumers to "essentialize" electronic content as a community and create a ranking system for the summaries generated by members of the community so that people can easily identify and directly consume the relevant parts of electronic content through both personal and crowd-sourced rating schemes.

SUMMARY

[0005] Embodiments contemplate a variety of improved methods and systems for enriching and consuming electronic content on PCs and mobile devices.

[0006] Embodiments include methods and systems for enabling a community of users to receive electronic content, generate an essence layer file, which includes annotations and highlights to the electronic content mapped to overlay on top of the electronic content and publish the essence layer file with the electronic content to the Internet. [0007] Additional elements include providing for a personal and community rating system for the essence layer files generated by individual users. Further, elements include a plurality of importance levels for highlights, user libraries, a community stream for annotated/highlighted electronic content, and for generating brief summaries of electronic content.

BRIEF DESCRIPTION OF THE FIGURES

[0008] These and other objects, features and characteristics of the present invention will become more apparent to those skilled in the art from a study of the following detailed description, in conjunction with the appended claims and drawings, all of which form a part of this specification. In the drawings:

[0009] FIG. 1 is a block diagram of a system for generating selection overlays;

[0010] FIG. 2 illustrates the combination of electronic content with an essence layer file;

[0011] FIG. 3 is a flowchart illustrating community use of essence layer files,

[0012] FIG. 4 is a flowchart for a number of stages of essence layer fi le use;

[0013] FIG. 5 illustrates example electronic content document formats;

[0014] FIG. 6 is a pictorial illustration of multimedia content with an anchored essence layer file;

[0015] FIG. 7 is a pictorial illustration of multimedia content with an anchored essence layer file limiting the view of the original multimedia content;

[0016] FIG. 8 is a pictorial illustration of multimedia content with an anchored essence layer condensing and formatting the view of the original multimedia content;

[0017] FIG. 9 is a pictorial illustration of essenced multimedia content sorted by creation date and content tags;

[0018] FIG. 10 is a pictorial illustration of essenced content organized into an essence library;

[0019] FIG. 1 1 is a pictorial illustration of condensed and reformatted essenced electronic content displayed in the essence library;

[0020] FIG. 12 is a pictorial illustration of an essence layer file anchored to electronic content and integrated with social features;

[0021] FIG. 13 is a pictorial illustration of essence layer file user controls;

[0022] FIG. 14 is a pictorial illustration of a web browser tool user interface; [0023] FIG. 15 is a pictorial illustration of a web browser tool bookmark bar control;

[0024] FIG. 16 is a user interface illustration for essenced video and audio content;

[0025] FIG. 17A is a pictorial illustration of essenced video content;

[0026] FIG. 17B is a pictorial illustration of a selected portion of essenced video content;

[0027] FIG. 18 is a pictorial illustration of essence layer file user controls to limit view of video content;

[0028] FIG. 19 is a pictorial illustration of video content with an anchored essence layer file limiting the view of the original video content;

[0029] FIG. 20 is a pictorial illustration of a web browser tool user interface for video content; and

[0030] FIG. 21 is a pictorial illustration of a web browser tool user interface for video content to share essence layer files. DETAILED DESCRIPTION

[0031] At the core of the essentializing system is the essence concept. An "essence" always is based on an "existing electronic content" and is created by a person (the "essencer" or "creator") who is analyzing said existing content and who is defining a "set of annotations," a "purpose message," an "identifying image," and a "list of tags." An essence can also have an "essence discussion stream. " Several essences can be grouped together into a "collection" of essences. Essences can be generated for numerous types of electronic content, such as HTML (web pages), PDFs, eBooks (ePub), videos (YouTube, MP4, etc.), audio recordings (MP3, streaming formats, etc.), text documents, Emails, virtual reality presentations, and other suitable electronic document formats. For the purposes of this disclosure, the term "essencer" refers to a person creating an essence. An essence has a link to the essencer.

[0032] FIG. 1 is a block diagram of a system 30 for generating selection overlays.

The system 30 includes a web server 32, a user device 34, and electronic content 36 that is often found on the Internet 38, though it may reside within the user device 34. The web server 32 is processor 40 operated and includes digital storage 42, storing server application software 44, and an essence library 46. The server application software 44 communicates with the user device 34 and provides a backend for user activities. The essence library 46 stores essence layer files, generated by user's index, by user and electronic content 36. [0033] The user device 34 includes a number of components and is processor 40 operated. In some embodiments, digital storage 42 includes client application software 48 and/or a web browser tool 50. A web browser tool 50 refers to browser plugin applications or a program code that is executed by a web browser. The user device 34 also includes a display 52, a user interface 54, and a network transceiver 56 for communicating with the web server 32 and the Internet 38.

[0034] Referring now to FIG. 2, a combination of an original document with an essence layer file is illustrated. An original document, electronic content 36, is combined with an essence layer file 58 to create an essenced document 60. The essence layer file 58 is an overlay file comprising a set of annotations 62. An "annotation" 62 refers to an element of electronic content (e.g., a piece of text, an image, or a piece of video) identified by a range or position in the electronic content 36, and can contain one or more of the following: A highlight marking the content (the highlight can have different styles, i.e. background colors, borders, etc.), an importance level that ranks the annotation relative to other annotations, levels for "agreement" and "disagreement" and also for "do not understand," a multi-media comment created by essencer (user who generates the essence layer file 58), labels, comments, and links to web resources, external documents, other annotations 62, or other essenced documents 60. An essence layer file 58 can be applied and stripped from an original document, electronic content 36, according to viewer specifications. Further, multiple essence layer files 58 can be combined on top of a single electronic content 36.

[0035] An essenced document 60 includes the essence annotations 62, along with the original document, electronic content 36. Highlighted content in annotations 62 depends on the kind of original document, electronic content 36, essenced. Highlights may be text, pictures, audio clips, video clips, or other suitable multimedia content known in the art. In some embodiments, essenced documents 60 would further include a purpose message created by the essencer. The purpose message may be a multi-media comment. Such a comment can also contain links to web resources, external documents, annotations in the essence layer file 58, or other essence layer files 58. In some embodiments, essenced documents 60 include an essence discussion stream 64 and sorting tags.

[0036] FIG. 3 is a flowchart illustrating public use of essence layer files 58 as generated pertaining to given electronic content 36 and created by an essencer. In step 302, a user selects electronic content 36, In step 304, the user activates the essencing system 30 for that electronic content 36. [0037] The same electronic content 36 often exists in multiple embodiments, which can either be in multiple documents 36 (e.g., as PDFs with file name A and with file name B), in different content formats (e.g., as HTML and as PDF), in different device form factors (e.g., as HTML for Desktop and for smartphones), under different URLs, or embedded in other different content (e.g., the same news text can appear in a web page of newspaper 1 and of newspaper 2). In step 306, using machine learning, optical character reading, text matching, and video data image matching, the server application software 44 can identify and match documents in multiple embodiments. In step 308, once matched, the electronic content 36 is sorted together with other matching electronic content 36 in the essence library 46. When the essence library 46 is searched by content, matched content appears together, anchoring existing annotations in various document formats. In step 3 10, annotations 62 selected by the user/essencer are anchored to the electronic content 36. Anchoring of annotations is performed differently depending on the style of electronic content 36 the user is viewing.

[0038] FIG. 4 is a flowchart for a number of stages of essence layer file use. In step

402, the web browser tool 50 or client application software 48 receives electronic content 36, where electronic content 36 is hosted on the Internet at a particular URL and includes any of: multimedia articles, streaming videos, podcasts, and text articles. In step 404, the web browser tool 50 or client application software 48 generates an essence layer file 58 which is associated with the particular URL and a given user/essencer. The essence layer file 58 comprises user controls and annotations 62 for the electronic content 36. The annotations 62 comprise highlights or comments to the portions of the electronic content that the given user deems of importance. The user controls enable the given user to limit the display of the electronic content 36 through the essence layer file 58,

[0039] In step 406, the web browser tool 50 or application software 44, 48 anchors the essence layer file 58 to the electronic content in the web browser such that the essence layer file 58 overlays on top of the electronic content 36 and annotations 62 match up to the portions of the electronic content 36 that a given user deems of importance. The anchoring is performed through formatting algorithms working over pre -formatted content (e.g., HTML), or content API's from external content publishers (e.g., YouTube).

[0040] In step 408, the essence layer file 58 anchored on the electronic content 36 limits the display of the electronic content 36 through the essence layer file 58, as displayed by the web browser or client software 48. In step 410, the web server 32 publishes the essence layer file 58 to a searchable essence library 46 indexed by user and organized by electronic content 36.

[0041] The essencer has additional editorial controls over the generated essence, and may save the essence in a personal essence library 46. In step 412, the essencer can further share the essence with others. Thus, the essenced document 60 can be shared over social networks, email, messaging applications, direct links, personal channels (such as a YouTube channel), or other suitable methods to disseminate electronic content known in the art. The essenced document 60 is shared with consumers who then read, watch, and analyze the essenced document 60 and may provide a rating for the specific essence, or essences, in case there are multiple essences layered on a single original electronic content 36. In step 414, the web server 32 accepts essence ratings for essence layer files 58 from community users. Higher ranked essence layer files 58 would be placed higher in search results such that more popular essencers would receive a greater number of views and the community as a whole would receive the highest quality summaries of the content that mattered to them.

[0042] The rating system could allow for personal rating to matter more than average community ratings such that a given consumer would be able to tailor the experience to that consumer's particular preferences. Further, different kinds of content could be categorized separately into sub-content such that community rating pertained to a relevant sub- community viewing the given content. One way of distinguishing sub-content is through the use of subject matter tags. An illustrative example of sub-content would be U.S. court opinions, and legal blogs; while the sub-community would consist of lawyers and legal professionals who cared about concise summaries of said opinions and blogs.

[0043] Referring now to FIG. 5, FIG. 5 illustrates essenced and original document formats, FIG. 5 is self-explanatory.

[0044] FIG. 6 is a pictorial illustration of multimedia electronic content 36 with an anchored essence layer file 58. The essenced HTML page 60 displays multiple levels of essence annotation 62. Examples of visual highlighting include: underlining, adjusting font size and weight, and highlighting of various colors. Annotations come in multiple, graduated importance levels. Each progressive level uses increasingly conspicuous annotations. Essencers are encouraged to properly use importance levels through the rating system. Essencers who create questionable or misleading annotations 62 or importance levels are down-rated and their essence layer files 58 will see less circulation.

[0045] FIG, 7 is a pictorial illustration of multimedia electronic content 36 with an anchored essence layer file 58 limiting the view of the original multimedia electronic content 36. Similarly to FIG. 6, essenced HTML documents 60 appears to consumers with the essence layer file 58 overlaid on the original electronic content 36; however, the client application software 48 or web browser tool 50 can be configured to obscure or mute all content that has not been annotated. In some embodiments, obscuring non-annotated content involves overlaying a color filter on top of the non-essential content. In an essenced ePub, chapters that did not have any annotations would also be obscured in the table of contents. The transistion between full view and obscured view can be performed with a smooth gradual animation altering states.

[0046] In using the web browser tool 50, in one embodiment, the essence layer file 58 is made at least partially opaque, and the elements, pictures or text selected for annotations 62 is merely copied to the opaque layer over the exact location of the original elements. In this way, the view of the electronic content 36 is limited. In another implementation, the original content 36 itself is altered in order to make the necessary appearance changes. In this implementation, there are not multiple layers, but rather amended code. Additional user controls enable a user to select an importance level, changing the importance level causes all annotations 62 below that importance level to be obscured or limited in the same fashion non- annotated elements are.

[0047] FIG. 8 is a pictorial illustration of multimedia electronic content 36 with an anchored essence layer file 58 condensing and formatting the view of the original multimedia electronic content 36. Rather than obscuring non-annotated content, in a formatted view 60A, the non-annotated elements simply would be removed and the display of annotated elements reorganized in a condensed fashion. In some embodiments, to achieve this effect, the essence layer file 58 is made entirely opaque, and annotated elements 62 are copied to the essence layer file 58 and formatted centrally in the web browser screen or the reader software (as appropriate). In other embodiments, the original content code is altered to hide ail non- annotated elements. Importance levels work similarly as above. Transition between difterent view states is performed with an animation of elements appearing and hiding gradually. An important feature of this view is that the basic document and paragraph structure is retained despite the fact that large parts of the original content 36 may be hidden.

[0048] FIG. 9 is a pictorial illustration of essenced multimedia content 60 sorted by creation date and content tags 66. In some embodiments, filters are generated from certain user-selectable criteria (e.g., subject tag search string and the creating user) and sorted by several user-selectable sorting criteria, such as the creation date. Users of the feed of essenced content would be presented with a plurality of essenced content stored in an essence library 46. The essence library 46 presents consumers with essenced content attuned to that consumer's own personal preferences and ratings. If multiple essence layer files 58 on given content were available and rated highly enough, the essence library 46 could optionally present these as a single entry. Each entry would contain either the essencer's written purpose statement, or only those annotations with the highest level of importance. Additionally, this essence library 46 would show tags associated to the essences. The essence library 46 can also be searched with a search bar 68 and allows for offline viewing of essenced content 60.

[0049] Referring now to FIG. 10, FIG 10 is a pictori al illustration of essenced HTML content 60 accessed from an essence library 46. When accessing essenced content 60 from the essence library 46, a consumer is provided with library controls 70 on the top of the user interface. These library controls 70 allow for operations such as filtering essenced content according to importance levels, tags, or rating essencers. Clicking on one of the references to an essence layer file 58 on the essence library 46 display redirects the web browser to the URL of the electronic content 36 with an essence layer file 58 overlaid.

[0050] FIG. 11 is a pictorial illustration of condensed and reformatted essenced electronic content 60 displayed in the essence library 46 through a web application 44, 48. When viewing the essence library 46 or a subset thereof in a web browser tool 50, web application 44, 48, or another suitable content reader tool, particular essence layer files 58 may be selected and expanded to the formatted view 60A (similar to FIG. 8).

[0051] FIG. 12 is a pictorial illustration of an essence layer file 58 anchored to electronic content 36 and integrated with social features. In one embodiment, when an essence layer file 58 is shared and a recipient (consumer) navigates to the shared link, the sharing view of an essence layer file 58 is opened. At the top of the sharing view is the sharing banner 64A, which includes an image of the essencer (or an avatar image) 64B, a sharing message 64C, which commonly would be the purpose message of the essence layer file 58 but which can be modified, and finally the sharing banner could contain links 74 to external resources or to annotations in the content (as shown in the image above). The view controls 76 allow the consumer to select one a level of view limitation on the electronic content 36 through importance levels, or as shown in FIG. 6 through FIG. 8. An optional discussion stream can be shown or hidden and is usually located at the bottom or the right side of the page (not pictured).

[0052] FIG. 13 is a pictorial illustration of annotation user controls 78. When using the web browser tool 50 or application software 44, 48, selecting text or other elements of electronic content brings up the annotation user controls 78. Annotation controls 78 include buttons to generate a number of levels or styles of annotation 62. Pictured here are three levels of highlighting, text comments, and delete annotation buttons.

[0053] FIG. 14 is a pictorial illustration of a web browser tool user interface 80.

Within essenced content 60 displayed in a web browser, the web browser tool 50 includes a web browser tool user interface 80 which controls the essence layer file 58. The example shown in FIG. 14 includes a tool bar with a number of buttons that cycle between importance levels/electronic content viewing limitation levels. Additionally included are sharing buttons and drop-down menus for additional controls.

[0054] FIG. 15 is a pictorial illustration of a web browser tool 50 bookmark bar control 80A. The web browser tool user interface 80 may include a book mark bar control 80A. In order to activate or create an essence layer file 58, one method includes the use of a bookmark bar control 80A. In use, while browsing in a web browser, a user may tap the bookmark bar control 80A and an essence layer file 58 is generated and overlaid on the currently viewed electronic content 36.

[0055] There are a number of ways to accomplish this. In some embodiments, a web browser plugin is activated upon clicking the bookmark bar control 80A. In some embodiments, clicking the bookmark bar control 80A captures the current URL 82 of the web browser and inserts the current URL 82 into a web application 44, 48 with a different URL that generates a new web page based on the data from the current URL 82, and further includes an essence layer file 58 overlaid on the current URL's 82 data.

[0056] Referring now to FIG. 16, FIG. 16 is a user interface illustration for essenced video and audio content as displayed in an essence library 46. Video or audio content would also be displayed in the essence library 46. In some embodiments, the entry would contain basic information included in an essence, as well as a drop-down pane, which would include individual annotations and nested annotations, all of which could be played individually.

[0057] An integration of an essence layer file 58 viewing application with external applications and reading devices can be very useful for multiple purposes, for example, in order to display essence layer file 58 information in the external applications or reading devices, or to open content that is loaded in external applications in the presently invented system in order to create or consume essenced content 60 and execute other functions of integration. Accordingly, the essence viewer would integrate with external applications and devices such as Evemote; email clients like Outlook and Mozilla Thunderbird; Microsoft Office applications like Word, eBook Reader apps and devices; web browsers; and other suitable applications known in the art. [0058] Referring now to FIG.s 17A and 17B, where FIG. 17A is a pictorial illustration of essenced video content 60 and FIG. 17B is a pictorial illustration of a selected portion of essence video content. Moving the mouse over the list illuminates the time frame of the annotation 62. Clicking on an annotation 62 starts play back of that portion. Clicking on "Play ail highlights" plays back the highlighted parts of the video; all non-highlighted parts are either not played at all or are played back in very high speed (fast forward). In this way, the total video content 36 is limited in view as only the highlighted annotations 62 are played back. Furthermore, user can filter the list according to importance levels of annotations 62, thus allowing the user to play only the most important parts. Audio recordings are similar to videos in that the individual audio pieces are highlighted. The functionality is the same.

[0059] Creating essences for videos can be a cumbersome and time-consuming task, so quick hot controls are established to aid in the task. In some embodiments, a key function is positioning the current video cursor and setting start and end points of annotations. When the essencer watching the video reaches a part in the video for which he/she wants to create an annotation, then the essencer needs to find the correct start and end points for the annotation. This may require going forth and back in the video several times to identify the ideal start point. Some embodiments support this special need by offering the keyboard and mouse/touch control functions in Table J below.

TABLE 1

Function Key Mouse / Touch

in control area

[0060] When the cursor or the left or right annotation handles are moved by mouse/touch or by keyboard, then the times are temporarily displayed above the

corresponding element. The essencer may also create temporary markers. While watching the video, the essencer might want to mark interesting positions in the video in order to create annotations later on. The user interface offers a function for setting temporary markers at the current cursor position, which could be done simply using the "new annotation" function and not setting an end.

[0061] Navigating essenced video content would also have special controls. Sample controls are provided in Table 2 below.

TABLE 2

[0062] Once created, essenced video content may include an identifying image or animated image. Each annotation 62 can have one or more images that serve as identifying images. For a given annotation, the user interface automatically suggests several images from which a user can select one or more identifying images. The selected images would be automatically transformed into a minimal size image (in case of a single image) or into an animated GIF or another animated image format (in case of multiple images). Additionally, the user interface allows the essencer to choose the identifying image for the essence by offering options for the selection. Such options include: one single static image out of all identifying images of all annotations, one animated image (e.g., animated GIF) out of the animated identifying images of all annotations, or an animated image (e.g., animated GIF) created of all identifying images of all annotations,

[0063] An additional use of annotations in essenced video or audio content would involve nested annotations. Nested annotations are those found within other, larger annotations.

[0064] FIG, 18 is a pictorial illustration of essence layer file 58 user controls to limit view of video content. Displayed is video content 36 taken via a video streaming service API when provided the URL of the hosted streaming video on that streaming service. Missing from the original video API (identified by the URL)are the rest of the host streaming service's controls, and social features. In this way, view of the original content is limited. Controls included in the figure include social controls 64, annotations 62, and essence layer file controls SOB that control limitation levels of viewing.

[0065] FIG. 1 is a pictorial illustration of video content with an anchored essence layer file 58 limiting the view of the original video content. When video content is shown through a formatted view 60B, a video's playback bar 83 includes annotations 62 that have certain durations. When played back in a formatted view 60B, only the annotations 62 are played back for the predetermined duration of each. Thus, a video timer 84 may be reduced from 20+ minutes to 2 minutes in duration. Additionally, the user is enabled to filter annotations below a certain importance level thereby playing a subset of the annotations and resulting in an even shorter playback,

[0066] FIG. 20 is a pictorial illustration of a web browser tool 50 user interface for video content. In some embodiments, rather than keyboard controls, graphical controls are used to create annotations 62 for videos. Included in these controls is an add highlight annotation button 86. The add highlight annotation button 86 creates the beginning of an annotation at the present progress of the video or audio data. There are also step controls 88 to move the progress of the audio or video data a specified amount. Alternatively, the progress may be moved on the progress bar 83 manually. Once the user has determined the chosen length of the annotation 62, the end annotation button 90 is pressed, and an end point to that annotation 62 is created. If no end is set for annotations then these annotations have a zero length and are jump marks helpful for defining a table of contents of videos, for example.

[0067] An additional feature not shown in the figure is the support for the automatic setting of annotations based on pauses or based on certain words in the audio track of a video. An analysis algorithm can be activated to set these marks by analyzing the audio stream of the video,

[0068] The video player of the system allows play a video in scanning mode, i.e. playing all marks for a certain amount of time and then automatically jumping to the next mark. E.g. a user can define that he/she wants to scan a video in a time of 30 sec. In that case the 30 sec would be split among all marks. E.g. if a video has 10 marks it would play each mark for 3 sec before jumping to the next mark.

[0069] This scanning even works with videos without any marks set. In such a case the video player automatically sets evenly distributed marks and would play them for a certain amount of time. The System suggests the number of automatic marks and a time lengths (allowing the user to adapt these values).

[0070] An optional component of the system will also offer the automatic audio analysis of videos and audio recording and video analysis of video recordings. This programmatic audio analysis can: detect pauses and spoken language sentences segments, and analyze voice to create automatic transcripts of spoken language in multiple languages. The system then offers to automatically set annotations based on pauses and sentence segments. The application software 44, 48 also offers to set marks based on certain keyword matches or sentence fragments detected in the automatic transcript.

[0071] An essence layer file 58 is enabled to direct the application software 44, 48 to create an automatic suggestion for annotating/essentializing videos and audio recordings based on such audio analysis results. Likewise the same offering will be available for automatically setting annotations based on video analysis. The application software 44, 48 creates new video data based on the annotations (highlights) in an existing video. For essences layer files 58 with highlights, just the highlights of the original video are copied to the new video. The rest of the video is cut out. For videos with marks, the user can select the length after each mark that should be copied to the new video. The rest of the video is cut out. Thus the new video is a short version based on the annotations of the original content.

[0072] FIG. 21 is a pictorial illustration of a web browser tool 50 user interface for video content to share essence layer files 58. A video can be shared by pushing the share button, a social control 64, which allows copying the sharing link (to be sent by email or posted on a social media). Also supported is sharing a video directly on external social media (e.g., Facebook, Twitter, Google+, and Linkedln).