BACKGROUND

[0001] This specification generally relates to providing context-based user interface components allowing a user to input text obtained in a first application into a user interface of a second application.

[0002] A user of a user device submits search queries to a search engine (or another search application) seeking information about different topics and in response, the search engine (or the other search application) provides search results that may be responsive to those topics. A search result is data generated by a search engine that identifies a resource that satisfies a particular search query, and includes a resource locator for the resource. An example search result can include a web page title, a snippet of text extracted from the web page, and the URL of the web page.

[0003] In some instances, when a user is using an application, he/she may want to utilize some information retrieved in response to a prior search query regarding a particular topic.

In such instances, unless the user recalls that information, the user will generally need to execute another search query for that same topic in a separate search environment, copy information included in the retrieved search result's, and then return to the original application to manually insert the copied information.

SUMMARY

[0004] In general, one innovative aspect of the subject matter described in this specification can be embodied in methods that can include the operations of storing, in a cache on a user device, structured data from results in response to a query, wherein: the structured data from the results include a set of attributes and a corresponding set of attribute values for an entity referenced in the query; and the structured data from the results were obtained in an instantiated search environment displayed in a first application; processing context data being displayed in a second application executing on the user device, wherein the second application is separate from the first application in which the instantiated search environment was displayed; determining, from the context data, a candidate entity and a candidate attribute of the candidate entity; identifying, using the structured data from the results, an attribute value for the candidate attribute of the candidate entity; generating a user interface component for the user to input text into the second application, wherein the input text includes the identified attribute value for the candidate attribute; and providing the user interface component in the second application for presentation on the user device. Other embodiments of this aspect include corresponding systems, devices, apparatus, and computer programs configured to perform the actions of the methods. The computer programs (e.g., instructions) can be encoded on computer storage devices. These and other embodiments can each optionally include one or more of the following features.

[0005] In some implementations, processing context data being displayed in the second application executing on the user device, can include determining that the context data does not include an attribute value for the candidate attribute of the candidate entity; and in response to determining that the context data does not include an attribute value for the candidate attribute of the candidate entity, identifying, using the structured data , the attribute value for the candidate attribute of the candidate entity.

[0006] In some implementations, the structured data that was obtained in the instantiated search environment can include structured data that was displayed in the instantiated search environment on the user device.

[0007] In some implementations, the structured data that was obtained in the instantiated search environment can include structured data that was (1) displayed in the instantiated search environment on the user device and (2) retrieved, but not displayed, in the instantiated search environment.

[0008] In some implementations, the context data can include text inserted by the user within the second application and the content being displayed in the second application on the user device.

[0009] In some implementations, the data stored in the cache can be limited to: (1) storing structured data for a threshold number of entities; or (2) storing structured data for a threshold amount of time.

[0010] In some implementations, providing the user interface component in the second application for presentation on the user device can include generating a user-selectable user interface component that includes the recommendation; and presenting the user interface component within the second application. [0011] Particular embodiments of the subject matter described in this specification can be implemented to realize the following advantages. The innovations described in this specification provide an improved technique for a user to input information obtained in a first application into a second application and improve the usability of the user device in comparison to conventional techniques. Conventionally, a user would have to copy information included in results retrieved in response to a query in one application, switch to another application, and manually input the copied information in the other application. In some instances, a user may need to repeatedly switch between applications until all the desired information has been copied from one application and input into the other application. In contrast, the techniques described in this specification avoid such repeated context switching (e.g., switching from one application to another) by storing the information included in the results retrieved in one application and making the stored information available to the user when using the other application (e.g., in the form of recommendations for text for the user to input into the other application). For example, if a user device submits a search query for a particular entity (e.g., a restaurant), the search results (e.g., address, opening time, etc.) can be stored in a cache and then a user interface component can be provided in a different application to the application in which the search results are obtained to allow this stored information to be input by the user in a different application (e.g., a maps application can recommend the user to input the address of this entity into this application and if the user selects this recommendation, the address can be inserted automatically into this application).

[0012] The innovations described in this specification improve the efficiency of the computing resources utilized to perform and retrieve searches in comparison to conventional techniques. Conventionally, a user device may need to utilize its computing resources to execute multiple search queries on the same topic, which in turn causes search platforms to utilize computing resources to retrieve the same search results that were already retrieved in response to a previous query. In contrast, the techniques described in this specification avoid expending computing resources to re-execute same queries and retrieve similar search results. Rather, the techniques described in this specification enable a user device to store the search results retrieved in response to a search query and subsequently make these search results available when a user of a user device may need this information in the future. For example, in one implementation described in this specification, a user device can store search results retrieved in response to a search query (e.g., information regarding a particular restaurant) and then make these search results available (e.g., in the form of recommendations for text input) to the user of the user device, instead of the user having to re-execute the same search query.

[0013] The details of one or more embodiments of the subject matter described in this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, the drawings, and the claims.

BRIEF DESCRIPTION OF THE DRAWINGS [0014] Figure 1 is a block diagram of an example environment 100 in which context- based text recommendations are provided to the user based on previously retrieved and stored search results.

[0015] Figure 2 is a block diagram of a system that generates context-based text input recommendations based on previously retrieved and stored search results.

[0016] Figure 3 is a flow diagram of an example process 300 for generating context- based text input recommendations based on previously-retrieved and stored search results. [0017] Figure 4 is a block diagram of an example computer system.

[0018] Like reference numbers and designations in the various drawings indicate like elements.

DETAILED DESCRIPTION

[0019] This specification relates to relates to providing context-based user interface components allowing a user to input text obtained in a first application into a user interface of a second application.

[0020] As described below and in detail throughout this specification, a content manager implemented on a user device can store structured data from search results that are retrieved in response to a user’s search query and can use this structured data to provide recommendations for text to input into an application based on the content displayed in the application. [0021] The content manager stores, in a cache on the user device, structured data from the search results that are provided within an instantiated search environment, in response to a search query. As used in this specification, structured data provide a list of attributes and corresponding values for an entity specified in the search query. For example, in response to a query for “Kevin Durant,” the structured data may include attributes associated with “Kevin Durant,” such as Salary, Team, Birth Year, Family, etc., along with answers that provide information (or values) about these attributes. The structured data may be extracted from a knowledge base (that stores attributes and attribute values for different entities) or may be extracted from search results retrieved in response to the query.

[0022] The content manager uses the stored structured search data to generate text recommendations for a user of the user device to input into a second application, which is generally different from the first application within which the search environment was instantiated. When a user accesses the second application, the content manager processes the context displayed within this application and uses the processed context to determine, e.g., using a statistical model (such as a supervised or unsupervised machine learning model), a candidate entity and a candidate attribute of the candidate entity. The content manager uses the candidate entity and the candidate attribute of the candidate entity to identify, from among the structured data stored in the cache, an attribute value for the candidate attribute. The identified attribute value is used by the content manager in generating recommendations for text for the user to input into the second application. These recommendations are provided, e.g., in the form of user-selectable user interface components (e.g., buttons) that the user can select and upon such selection, the recommended text is inserted within the portion of the second application in which the user inputs text (e.g., using a keyboard).

[0023] These and additional features are described in detail throughout the specification. [0024] Figure 1 is a block diagram of an example environment 100 in which context- based text recommendations are provided to the user based on previously retrieved and stored search results.

[0025] The example environment 100 includes a network 106, such as a local area network (LAN), a wide area network (WAN), the Internet, or a combination thereof. The network 106 connects user devices 102 and search platforms 108. The example environment 100 may include many search platforms 108 and many user devices 102. [0026] A user device 102 is an electronic device that is capable of requesting and receiving content over the network 106. Example user devices 102 include personal computers, mobile communication devices, digital assistant devices, and other devices that can send and receive data over the network 106. A user device 102 typically includes a user application, such as a web browser, to facilitate the sending and receiving of data over the network 106, but native applications executed by the user device 102 can also facilitate the sending and receiving of content over the network 104. Examples of content presented at a user device 102 include webpages, word processing documents, portable document format (PDF) documents, images, videos, and search results pages.

[0027] A search platform 108 is a computing platform that identifies and provides to user devices 102, resources (e.g., links to web pages, applications, etc.) as well as information, e.g., in the form of structured search results that are responsive to search queries submitted by users of the user devices 102. Example search platforms 108 include search engines, social media platforms, news platforms, data aggregator platforms, or other content sharing platforms.

[0028] A user using an application (e.g., a web browser, a native application that enables web search) on a user device 102 can submit search queries to a search platform 108 and retrieve search results in response to the submitted searched queries. As described below with reference to Figures 2 and 3, a content manager 104 executing on the user device 102 can store the structured data from search results obtained from the search platform 108 and subsequently, use the stored structured data in providing context-based text input recommendations to the user of the user device 102.

[0029] Figure 2 is a block diagram of a system that generates context-based text input recommendations based on previously retrieved and stored search results.

[0030] As shown in Figure 2, the content manager 104, which is generally implemented on the user device 102, includes a context analyzer engine 230, a recommender engine 232, and a cache 234. The tasks performed by the content manager 104 and each of its components is briefly summarized in the following paragraph and described in detail with reference to Figure 3. Engine is also referred to as an apparatus in this specification, and generally includes software instructions executing on or more hardware units, such as processors. [0031] In some implementations, the content manager 104 is implemented in the operating system of the user device 102. In such implementations, when the user device 102 requests and retrieves search results from a search platform 108 using an application, e.g., a native application or a browser, the software development kit (SDK) of the application invokes an operating system API to store some or all of the structured data from search results in the cache 234. Subsequently, when the user of the user device 102 invokes another application, the application requests that the content manager 104 process the content displayed in the application using the context analyzer engine 230. Based on this content, the recommender engine 232 uses the search results stored in the cache 234 to generate recommendation/s for text for the user to input into the application.

[0032] Figure 3 is a flow diagram of an example process 300 for generating context- based text input recommendations based on previously retrieved and stored search results. Operations of the process 300 are described below for illustration purposes only. Operations of the process 300 can be performed by any appropriate device or system, e.g., any appropriate data processing apparatus (as shown and described with reference to Figure 4). Operations of the process 300 can also be implemented as instructions stored on a non- transitory computer readable medium. Execution of the instructions cause one or more data processing apparatus to perform operations of the process 300.

[0033] A user device 102 obtains search results in response to a search query (at 302). In some implementations, a user of a user device 102 enters a search query in an instantiated search environment within a first application. As used in this specification, a first application can be a web browser or another application executing on the user device 102 (e.g., a native application) within which a search environment can be instantiated, e.g., by navigating within a web browser to a search platform 108’s website or by launching native application that interacts with a search platform 108 in submitting search queries and retrieving results in response to search queries.

[0034] In response to the search query sent by the user device 102, the search platform 108 obtains search results and sends these search results to the user device 102. The search results can include search results, which include resources (e.g., links to web pages, applications, etc.) as well as information, e.g., in the form of structured search results that are responsive to the search query. [0035] Figure 2 shows an example of the search results retrieved and provided within the instantiated search environment of the first application, which in this example is a search platform 108's website accessed in a mobile web browser on the user device 102. As shown in the instantiated search environment 204, the search query 220 — “Bob's Burger” — generates a search results page, which includes the structured data 222 and the search results 224. The search results 224 are generally links to webpages or applications that are identified by the search platform 108 as responsive to the search query. On the other hand, the structured data 222 identifies different attributes for the entity specified in the query along with values for those attributes. An entity can be a person, place, organization, concept, etc., and an attribute is a string that describes some aspect or characteristics of the entity. Examples of attributes can include the “date of birth” of a person, the “population” of a country, the “salary” of the athlete, or the “CEO” of an organization. In the example shown in Figure 2, for the search entity — “Bob's Burger,” the structured search results provide attributes for this entity — such as Type, Address, Opening Time, Closing Time, and Chef, along with the values for these attributes.

[0036] In some implementations, although the structured data 222 provided by the search platform 108 may include multiple attributes (and corresponding values) for the entity identified in the query, the user device 102 or the first application may only present a subset of these attributes (and corresponding values) within the first application on the user device 102. For example, the search platform 108’s website may have display rules that specify that only a threshold number of attributes should be provided for presentation in the first application on the user device 102 (e.g., the first N attributes of the M total attributes, the top five highest ranked attributes (in which case rankings of the attributes may be provided by the search platform 108)).

[0037] The content manager 104 stores, in a cache 234 on the user device 102, structured data from search results that were obtained in the instantiated search environment on the user device 102 (at 304). The content manager 104 stores the structured data (e.g., the structured data 222) in the cache 234 of the user device 102. In some implementations, the content manager 104 stores, in the cache 234, a data record for each entity along with the attributes and corresponding attribute values obtained and provided by the search platform 108. As shown in Figure 2, a data record 236 is stored in the cache 234, which includes the name of the entity (Bob’s Burger) and the corresponding attributes and attribute values for this entity that were retrieved in response to the search query.

[0038] In some implementations, the content manager 104 can only store in the cache 234 those attributes (and their corresponding values) which are displayed in the instantiated search environment 222. In other implementations, the content manager 104 can store in the cache 234 attributes that are displayed within the instantiated environment 204 as well as those attributes (and corresponding values) that were obtained from the search platform 108 but not displayed within the instantiated environment 204 (e.g., due to certain display rules described above). As shown in Figure 2, the data record 236 includes additional attributes (such as Rating and Attire) that are not displayed in the structured data 222 that are displayed within the search environment 204 in the first application. Thus, as described above, the cache 234 stores structured data for entities retrieved in response to search queries performed on the user device 102.

[0039] Because the cache 234 has a finite amount of space, it is generally not possible (nor feasible) to store all the structured search results obtained in response to all of the search queries run by a user. Accordingly, in some implementations, the amount of data stored in the cache 234 can be limited or managed in different ways, as described below.

[0040] As one example, the amount of data stored in the cache 234 can be limited by only storing structured search results for a threshold number (e.g., 10, 15, 20) of entities. Thus, if data for a threshold number of entities is already stored in the cache 234, structured search results for another entity may only be stored in the cache 234 if an existing entity stored in the cache 234 (e.g., the earliest stored entity) is first removed/deleted from the cache 234. In such implementations, to identify the age of entities for which data is stored in the cache 234, a timestamp may be associated with each data record stored in the cache. [0041] As another example, the amount of data stored in the cache 234 can be limited by only storing structured data for a specified period (e.g., one day, one week, etc.). In such implementations, structured data is stored along with a timestamp, which specifies the day and-'or time when that data was stored. When the structured data has been in the cache 234 for the specified period (e.g., as determined by using the timestamp to calculate the time/days elapsed since the results were stored and then comparing this elapsed day/time to the specified period), the content manager 104 deletes (or overwrites) such structured data from the cache 234.

[0042] As another example, the amount of data stored in the cache 234 can be limited by not only storing structured data for a specified period, but also by limiting the number of entities for which structured data is stored in the cache 234. Other conventional techniques for managing data stored in caches can also be implemented in limiting the amount of data stored in the cache 234.

[0043] A user of the user device 102 accesses a second application (at 306). In some implementations, the second application is separate from the first application in which the instantiated search environment was displayed. The second application can be an application (e.g., a web application or a native application) executing on the user device 102, such as a messaging application, an email application, a map application, or another application. In some implementations, the second application can be the same as the first application, in which case, once the search results are stored in the cache 234, the second application does not execute the instantiated search environment. As shown in Figure 2, the second application 206 is an instant messaging application (labeled “Messenger”).

[0044] The content manager 104 determines a candidate entity and a candidate attribute of the candidate entity based on processing the context displayed in the second application (at 308). In some implementations, while the user device 102 is executing the second application, the context analyzer 230 executes a script that processes the context being displayed in the second application. Using Figure 2 as an example, the script executing in the second application 206 processes the content of the messages (shown in the display area 248) as well as the content entered in the keyboard input area 246. In some implementations, the context analyzer 230 can also process characteristics about the application (e.g., the type of the second application (such as whether it is a maps application, an instant messaging application)).

[0045] Using the context processed by the second application, the context analyzer 230 determines a candidate entity and a candidate attribute of the candidate entity. As used in this specification, the candidate entity and the candidate attribute of the candidate entity refer to an entity and its attribute for which the recommender engine 232 will generate a recommendation for text for the user to input into the second application when the candidate entity corresponds to a data record for an entity in the cache 234.

[0046] The context analyzer 230 can include a supervised or an unsupervised machine learning model, e.g., a natural language processing model or a neural network model (or another appropriate statistical model) that identifies the candidate entity and the candidate attribute using the processed content. In some implementations, the context analyzer 230 can include a supervised machine learning model that is trained using a set of training data including processed context from different resources, such as web pages, documents, etc. (including content from these resources as well as the characteristics about the second application) and corresponding labels specifying the known candidate entity/ies and candidate attributes for these resources. Once trained, this machine learning model accepts as input the content processed by the context analyzer 230 and outputs the candidate entity/ies and corresponding candidate attributes referenced in the content. Using Figure 2 as an example, the machine learning model may determine from the obtained content that the candidate entity is “Bob’s Burger” and the candidate attribute of this entity is “Open Time.” As another example, the machine learning model may determine from the processing characteristics about a maps application and determine that the candidate entity is “Bob’s Burger” and the candidate attribute of this entity is “Address.”

[0047] In some implementations, the context analyzer 230 only identifies those candidate attributes for which the context of the second application does not already provide an attribute value. For example, if the context displayed in the second application 206 included the 5 P.M. opening time for Bob’s Burger, then the context analyzer 230 would not identify the open time as a candidate attribute. In such implementations, the machine learning model can be trained to avoid identifying a candidate attribute for which the content displayed in the second application already includes an attribute value for this attribute. The machine learning model can be trained using the set of training data that includes context from resources (e.g., web pages, documents, etc.), in which the context includes both attributes with attribute values and attributes without any attribute value, and labels that identify as candidate attributes only those attributes for which values are not already present in the content within the resources. [0048] The content manager 130 identifies, using the structured data from the search results, an attribute value for the candidate attribute of the candidate entity (at 310). In some implementations, the context analyzer 230 sends the identified candidate entity and the candidate attribute of the candidate entity to the recommender engine 232. The recommender engine 232 uses the candidate entity (e.g., the name or other identifier of the candidate entity) to determine whether a data record for an entity in the cache 234 has the same (or similar) name or identifier as the candidate entity, and if so to identify the data record for an entity in the cache 234 with the same name or the same identifier as the candidate entity. Using the example shown in Figure 2, the recommender engine 232 searches for the candidate entity “Bob’s Burger” within the data records stored in the cache 234 until it finds the data record 236 for the entity “Bob’s Burger.”

[0049] If a data record exists for the identified candidate entity, the recommender engine 232 searches for the candidate attribute within the attributes included in the identified data record for the candidate entity. Using the example shown in Figure 2, the recommender engine 232 searches for the candidate attribute “Open Time” (or another appropriate identifier for this attribute) within the data record 236 for Bob’s Burger and identifies a matching attribute of Open Time. If the recommender engine 232 determines that the identified data record for the candidate entity includes the candidate attribute, then the recommender engine 232 retrieves the attribute value corresponding to this candidate attribute. Using the example shown in Figure 2, the recommender engine 232 retrieves the value (“5 P.M”) corresponding to the candidate attribute “Open Time” within the data record 236 for the entity Bob’s Burger.

[0050] In some implementations, the attribute stored in the cache 234 may not exactly match the candidate attribute, but the cached attributes may nevertheless include the value for the candidate attribute. For example, the cached attribute may be “Business Hours” with the value “5 P.M. to 11 P.M.” The recommender engine 232 can have rules that specify that certain stored attributes can include values for two or more sub-attributes. In the above example, the recommender engine 232 can include rules specifying that the string before the preposition “to” (or another delimeter such as hyphen) represents the Open Time and the text after the delimeter represents the Close Time. In such instances, the recommender engine 232 identifies the “Business Hours” attribute as relevant to the candidate attribute of “Open Time,” and then retrieves the stored value preceding the delimeter (which in this case is the preposition “to”), which in this case is 5 P.M. Thus, the recommender engine 232 need not find an exact match for the candidate attribute within the stored attributes of the cache 234. Rather, the recommender engine 232 can search for and identify an attribute type (e.g., an attribute that encompasses two or more sub-attributes) that is similar to the candidate attribute (i.e., based on similar characteristics, such as time, location; e.g., business hours and Open Time share a characteristic of time and have similar units) and then extract the relevant portion of the corresponding value stored in the cache 234 for that stored attribute type.

[0051] If a data record and attribute value are identified, the content manager 130 generates a recommendation for text for the user to input into the second application (at 312). In some implementations, the recommender engine 232 generates a user interface element corresponding to a recommendation for text, which includes the identified attribute value for the candidate attribute. In the above example, the recommender engine 232 generates a recommendation of “5 PM” as text that the user may want to input into the second application.

[0052] The content manager 130 provides the recommendation in the second application for presentation on the user device (at 314). In some implementations, the recommender engine 232 provides the recommendation text (as described above in operation 312) for display in the second application, e.g., in a recommendation panel 240, and provides a user interface component (e.g., a button) that allows the user of the user to select this recommendation. In the above example, the recommender engine 232 generates a recommendation of “5 P.M.” as text that is included as a button with the recommendation panel 240 shown in the second application 206. In some implementations, the user interface component showing the text recommendation is presented in a system-level component, such as a keyboard, that is presented within the second application. For example, the system keyboard may be activated when a user selects a text-input field of a second application, and the text recommendations may be provided on this keyboard as buttons (separate from the existing buttons of the keyboard) that the user can select.

[0053] The user of the user device 102 can select (e.g., using a mouse, by touch) the interface component that includes the recommendation text. If the user selects the interface component, the second application includes this recommendation text in the user entry portion 246. In the above example, if the user selects the button (e.g., on a system level component, such as a keyboard) with the recommendation of “5 PM,” this text is then included in the user entry portion 246 at the point where the cursor is currently positioned. Thus, in this example, because the cursor is located after the word “at,” the recommendation text of 5 PM is inserted at that point. Once inserted, the text in the user entry portion 246 reads “THEY OPEN AT 5 P.M.”

[0054] Therefore, as described with reference to Figures 1-3, the content manager implemented on the user device stores structured data from search results retrieved in response to user’s search query executed within a first application, and uses the structured data in generating recommendations for text for the users to input into the second application based on the context displayed in the second application.

[0055] Figure 4 is block diagram of an example computer system 400 that can be used to perform operations described above. The system 400 includes a processor 410, a memory 420, a storage device 430, and an input/output device 440. Each of the components 410, 420, 430, and 440 can be interconnected, for example, using a system bus 450. The processor 410 is capable of processing instructions for execution within the system 400. In some implementations, the processor 410 is a single-threaded processor. In another implementation, the processor 410 is a multi-threaded processor. The processor 410 is capable of processing instructions stored in the memory 420 or on the storage device 430. [0056] The memory 420 stores information within the system 400. In one implementation, the memory 420 is a computer-readable medium. In some implementations, the memory 420 is a volatile memory unit. In another implementation, the memory 420 is a non-volatile memory unit.

[0057] The storage device 430 is capable of providing mass storage for the system 400.

In some implementations, the storage device 430 is a computer-readable medium. In various different implementations, the storage device 430 can include, for example, a hard disk device, an optical disk device, a storage device that is shared over a network by multiple computing devices (e.g., a cloud storage device), or some other large capacity storage device. [0058] The input/output device 440 provides input/output operations for the system 400. In some implementations, the input/output device 440 can include one or more of a network interface devices, e.g., an Ethernet card, a serial communication device, e.g., and RS-232 port, and/or a wireless interface device, e.g., and 802.11 card. In another implementation, the input/output device can include driver devices configured to receive input data and send output data to other input/output devices, e.g., keyboard, printer and display devices 460. Other implementations, however, can also be used, such as mobile computing devices, mobile communication devices, set-top box television client devices, etc.

[0059] Although an example processing system has been described in Figure 5, implementations of the subject matter and the functional operations described in this specification can be implemented in other types of digital electronic circuitry, or in computer software, firmware, or hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them.

[0060] Embodiments of the subject matter and the operations described in this specification can be implemented in digital electronic circuitry, or in computer software, firmware, or hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions, encoded on computer storage media (or medium) for execution by, or to control the operation of, data processing apparatus. Alternatively, or in addition, the program instructions can be encoded on an artificially-generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal, that is generated to encode information for transmission to suitable receiver apparatus for execution by a data processing apparatus. A computer storage medium can be, or be included in, a computer-readable storage device, a computer-readable storage substrate, a random or serial access memory array or device, or a combination of one or more of them. Moreover, while a computer storage medium is not a propagated signal, a computer storage medium can be a source or destination of computer program instructions encoded in an artificially-generated propagated signal. The computer storage medium can also be, or be included in, one or more separate physical components or media (e.g., multiple CDs, disks, or other storage devices).

[0061] The operations described in this specification can be implemented as operations performed by a data processing apparatus on data stored on one or more computer-readable storage devices or received from other sources. [0062] The term “data processing apparatus” encompasses all kinds of apparatus, devices, and machines for processing data, including by way of example a programmable processor, a computer, a system on a chip, or multiple ones, or combinations, of the foregoing. The apparatus can include special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit). The apparatus can also include, in addition to hardware, code that creates an execution environment for the computer program in question, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, a cross-platform runtime environment, a virtual machine, or a combination of one or more of them. The apparatus and execution environment can realize various different computing model infrastructures, such as web sendees, distributed computing and grid computing infrastructures.

[0063] A computer program (also known as a program, software, software application, script, or code) can be written in any form of programming language, including compiled or interpreted languages, declarative or procedural languages, and it can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, object, or other unit suitable for use in a computing environment. A computer program may, but need not, correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files that store one or more modules, sub-programs, or portions of code). A computer program can be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and interconnected by a communication network.

[0064] The processes and logic flows described in this specification can be performed by one or more programmable processors executing one or more computer programs to perform actions by operating on input data and generating output. The processes and logic flows can also be performed by, and apparatus can also be implemented as, special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit).

[0065] Processors suitable for the execution of a computer program include, by way of example, both general and special purpose microprocessors. Generally, a processor will receive instructions and data from a read-only memory or a random access memory or both. The essential elements of a computer are a processor for performing actions in accordance with instructions and one or more memory devices for storing instructions and data. Generally, a computer will also include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g., magnetic, magneto-optical disks, or optical disks. However, a computer need not have such devices. Moreover, a computer can be embedded in another device, e.g., a mobile telephone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a Global Positioning System (GPS) receiver, or a portable storage device (e.g., a universal serial bus (USB) flash drive), to name just a few. Devices suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The processor and the memory can be supplemented by, or incorporated in, special purpose logic circuitry.

[0066] To provide for interaction with a user, embodiments of the subject matter described in this specification can be implemented on a computer having a display device, e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user and a keyboard and a pointing device, e.g., a mouse or a trackball, by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, speech, or tactile input. In addition, a computer can interact with a user by sending documents to and receiving documents from a device that is used by the user; for example, by sending web pages to a web browser on a user’s client device in response to requests received from the web browser.

[0067] Embodiments of the subject matter described in this specification can be implemented in a computing system that includes a back-end component, e.g., as a data server, or that includes a middleware component, e.g., an application server, or that includes a front-end component, e.g., a client computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the subject matter described in this specification, or any combination of one or more such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include a local area network (“LAN”) and a wide area network (“WAN”), an inter-network (e.g., the Internet), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks).

[0068] The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. In some embodiments, a server transmits data (e.g., an HTML page) to a client device (e.g., for purposes of displaying data to and receiving user input from a user interacting with the client device). Data generated at the client device (e.g., a result of the user interaction) can be received from the client device at the server.

[0069] While this specification contains many specific implementation details, these should not be construed as limitations on the scope of any inventions or of what may be claimed, but rather as descriptions of features specific to particular embodiments of particular inventions. Certain features that are described in this specification in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination. Moreover, although features may be described above as acting in certain combinations and even initially claimed as such, one or more features from a claimed combination can in some cases be excised from the combination, and the claimed combination may be directed to a subcombination or variation of a subcombination.

[0070] Similarly, while operations are depicted in the drawings in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. In certain circumstances, multitasking and parallel processing may be advantageous. Moreover, the separation of various system components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0071] Thus, particular embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. In some cases, the actions recited in the claims can be performed in a different order and still achieve desirable results. In addition, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In certain implementations, multitasking and parallel processing may be advantageous.