TO DETERMINE KEYWORD QUALITY

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims priority to and the benefit of United States utility patent application no. 14/601, 114, filed on January 20, 2015, entitled Processing And Analysis Of User Data To Determine Keyword Quality, which is incorporated herein in its entirety.

BACKGROUND

Aspects of the present disclosure relate to attracting potential customers to a Web site and, more particularly, to the use of data relating to user engagement with a Web site to identify keywords to attract more customers.

Companies with revenue models centering on online presence put a lot of effort into getting more customers to buy things at a company's Web site. In the Internet world, there is significantly more browsing than buying. Data on who actually bought something, and how purchasers are attracted to an online merchant to buy something, generally is too sparse to be meaningful, even for large companies.

In theory, spending enough advertising money online, for example, in bidding on keywords, could yield a significant site conversion rate (i.e. customers who actually buy something after visiting the Web site). However, spending limitless advertising dollars obviously is not a profitable business model.

It would be useful to be able to find a way to improve a company's site conversion rate without spending an excessive amount on online advertising.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 shows aspects of overall architecture of a system according to an embodiment.

FIG. 2 is a flow chart showing aspects of operation of the techniques described herein according to an embodiment.

FIG. 3 depicts aspects of user engagement activity according to various embodiments.

FIG. 4 shows aspects of an embodiment of hardware which may be used in one or more of the server(s) or other hardware depicted in FIG. 1.

DETAILED DESCRIPTION A system, method, and medium monitor and examine user activity on an online marketplace. Responsive to certain user engagement criteria, in one aspect decisions are made to purchase certain keywords on Web search engines, as part of a search engine optimization plan.

In the following description, the terms "Web site" and "Web presence" may be used in different places. Neither term is intended to be limited to a particular Web page. Rather, the terms are intended to refer to any Web pages that a company might have. Thus, for example, in an online marketplace, individual Web pages may come and go as sellers list products for sale, and either buyers buy them, or the time for purchasing the products elapses. References herein to a company's "Web site" or "Web presence" are intended to capture all such pages that would in some fashion involve the company's basic URL.

While much of the description herein is provided in the context of an online marketplace, the techniques and accompanying structure described herein are not so limited, but rather are applicable to any Web presence to which its operator wishes to drive traffic.

FIG. 1 is a network diagram depicting a network system 100 having a client-server architecture configured for exchanging data over a network, according to an embodiment. The data may pertain to various functions (e.g., online item purchases) and aspects (e.g. managing content) associated with the network system 100 and its users. Although illustrated herein as a client-server architecture, other embodiments may include other network architectures, such as peer-to-peer or distributed network environments.

A data exchange platform 102, for example, including a marketplace application 120, may provide server-side functionality via a network 104 (e.g., the Internet) to one or more clients. The one or more clients may include users that utilize the network system 100 and, more specifically, the marketplace application, to exchange data over the network 104. The utilization may include transmitting, receiving (communicating), and processing data to, from and regarding content and users of the network system 100. The data may include, but are not limited to, content data such as item attributes, pricing and descriptive information, item listings associated with buyers and sellers, and auction bids. The data may also include transaction data involving collection and payment, shipping transactions, and real time synchronization of financial journals, among others. In an embodiment, the network system 100 accommodates buying and selling of items in an online marketplace.

In various embodiments, data exchanges within the network system 100 may depend on user-selection functions available through one or more client or user interfaces (UIs). The UIs may be associated with a client machine, such as a client machine 110 using a web client 106. The web client 106 may communicate with the marketplace application 120 over network 104 via a web server 116. The UIs may also be associated with a client machine 112 using a programmatic client 108 such as a client application, or a third party server 130 with a third party application 128. It can be appreciated that in various embodiments, the client machines 110, 112, or third party server 130 may be associated with a buyer, a seller, a third party electronic commerce platform, a payment service provider, a shipping service provider, or a financial institution system, each in communication with the data exchange platform 102 and optionally each other. The buyers and sellers may be any one of individuals, merchants, or service providers.

Turning more specifically to the marketplace application 120, an application program interface (API) server 1 14 and a web server 116 are coupled to, and provide programmatic and web interfaces respectively to, one or more application servers 118. The application server 118 hosts one or more marketplace applications 120. The application server 118, in turn, may be coupled to one or more database servers 124 that facilitate access to one or more databases 126.

In an embodiment, the web server 116 and the API server 114 communicate and receive data pertaining to listings, transactions, and item attributes. For example, the web server 116 may send and receive data to and from a webpage on a browser application (e.g., web client 106) operating on a client machine (e.g., client machine 110). The API server 114 may send and receive data to and from an application (e.g., programmatic client 108) running on another client machine (e.g., client machine 112).

In an embodiment, the marketplace application 120 facilitates listings and price- setting for sellers and buyers to list, bid on, and/or buy items in the online marketplace.

Looking further at FIG. 1, in an embodiment keyword data is aggregated in aggregated keyword data server(s) 150, as will be known to ordinarily skilled artisans. In one embodiment, the aggregation engine in server(s) 150 employs Hadoop®, but other embodiments may use other suitable aggregation engines. Platform 102 can communicate site query data with aggregated keyword data server(s) 150. In an embodiment, the site query data may include data regarding user (potential customer) engagement with platform 102. Such data will be referred to herein as user engagement data. The nature and evaluation of such data will be discussed in greater detail herein.

It will be appreciated that the data that aggregated keyword data server(s) 150 provide will be historical data. This data will include information about a variety of types of user engagement activity. Some of this data may be more prevalent at certain times, or with respect to certain types of items or goods for which users are shopping, among other reasons. Prevalence of certain types of data (e.g. impressions, click-through rate) may have an effect on how much weight those types of data will have compared to others.

Search engine 160, which may be third party search engine such as Google® or Bing®, can provide a wide variety of data on paid search, via paid search data module(s) 162, to aggregated keyword data server(s) 150. Paid search data may include data such as revenue per click (RPC); cost per click (CPC), also referred to as pay per click (PPC); revenue per impression (RPI); cost per impression (CPI); and percent click through ratio (%CTR). Some search engines provide their own (subjective) data, sometimes called a quality score, relating to the quality and relevance of keywords and PPC advertisements. A search engine company may use the quality score to determine what to charge for advertising. With such search engines, improving keyword relevance can improve the quality score, and can decrease the cost of advertising.

As will be discussed in more detail herein, keyword evaluation server(s) 170, including training module 172 and evaluation module 174, uses data received from search engine 160 and aggregated keyword data server(s) 150 to select keywords on which to bid. Training module 172 and evaluation module 174 are shown as separate In FIG. 1, but they need not be separate.

Keyword evaluation server(s) 170 may receive various analytics, including search result information, from search engine 160, and may evaluate those analytics in the context of site engagement data to develop potential keywords. Keyword evaluation server(s) 170 also may use paid search data, either directly from search engine 160 or as processed by aggregated keyword data server(s) 150, to develop a keyword bidding strategy.

Training module 172 may receive aggregated keyword data from server(s) 150, including for example historical data, to develop a model that, in an embodiment, predicts a likelihood that a user attracted to a company's Web site will actually buy something and thus generate revenue for the company. Such revenue generation from customers visiting a Web site may be referred to as conversion, as that term was used earlier. The model learns weights of different types of data, and may assign weights in the model accordingly.

Evaluation module 174 may receive information about the conversion model from training module 172; aggregated keyword data from server(s) 150; and market-based feedback from search engine 160, and may generate, for example, value scores for keywords. Those value scores may be compared, for example, with information received from search engine 160 regarding what certain keyword bidding costs. The outcome of that comparison may result in the generation of one of more keyword bids. In an embodiment, information about keyword bidding budgets, either as an overall amount or as allocated for keywords for particular products or categories of products, may be used as part of the decision whether to generate keyword bids.

FIG. 1 shows aggregated keyword data server(s) 150 connected to platform 102 on the same network as search engine 160 and keyword evaluation server(s) 170. However, the structure and techniques described herein are not so limited. Server(s) 150 may be connected more directly to platform 102, or may be connected to platform 102 over a different network than network 104. Server(s) 150 may be connected to platform 102 over the same network as server(s) 170.

FIG. 1 shows keyword evaluation server(s) 170 connected to platform 102 on the same network as aggregated keyword data server(s) 150 and search engine 160. However, the structure and techniques described herein are not so limited. Server(s) 170 may be connected more directly to platform 102, or may be connected to platform 102 over a different network than network 104. Server(s) 170 may be connected to platform 102 over the same network as server(s) 150.

Looking generally at user interactions with a Web site, a potential customer (user) may come directly to a company's Web site by entering the company's URL into a browser. Alternatively, the potential customer may come to the company's Web site by clicking on one or more of the links posted on pages which are presented in response to the potential customer entering search keywords into a Web search engine. The links could be search results, or they could be parts of paid advertisements which are placed by bidding on keywords. Analytics can tell a company how a potential customer came to the company's site.

Once potential customers arrive at the company's Web site, the nature of the potential customers' interaction with the site can inform the company what might promote more traffic to the company's site. Item purchases, as a percentage of traffic at a company's Web site, generally are considered too sparse to be particularly informative. However, interaction with the Web site prior to purchase, even if the interaction does not result in a purchase, can provide more dense data, and more robust insight into what might attract customers to the Web site. In particular, the insight can lead to more intelligent selection of keywords on which to bid.

For popular, high sales volume products with multiple sales channels, numerous entities, including competitors, may be willing to bid on certain keywords. A company may wish nevertheless to bid on those keywords, to ensure that the company's Web pages show up more prominently in search results pertaining to those products.

For less popular products, competition may not be so intense. Numerous products, of course, fall into this category. By monitoring how customers are searching for and viewing such products on the company's Web site, the company can mine the resulting data and devise an enhanced keyword selection and bidding strategy.

Companies with online presence may receive virtually countless visits from potential customers. In order to turn potential customers into customers, it is helpful to analyze how a customer interacts with a Web site. As will be discussed herein, customer interactions can indicate higher or lower likelihood of success of potential keywords, based for example on queries (using, of course, various words and word combinations) which customers undertake on the Web site. For example, the more a potential customer interacts with a particular query - for example, viewing query results, looking at particular items in the results - the more likely it is that the words the customer used to compile that query indicate the suitability of the keywords in the context of the Web site.

Customer queries are not a perfect source of data. Some customers may interact extensively with queries that they ultimately discard. For example, as a result of viewing a sufficient number of results of a particular query, a customer may learn that the query may need to be changed - for example, to use fewer keywords; to use the same keywords in different relationship to each other; to use different keywords; or any combination of these three possibilities. This list is not intended to be exhaustive; other query changes will be apparent to ordinarily skilled artisans.

If a customer spends very little time, or relatively little time with the query, that more limited interaction may inform the evaluation engine that the user found the query, or its contents, unsuitable. If the customer spends a relatively long time with a particular query, and then discards it in favor of another with different keywords, that kind of interaction can provide mixed signals to the evaluation module 174. On the one hand, the customer did do a lot with the query, indicating suitability of the keywords in the query. On the other hand, the customer did discard the query in favor of another, possibly indicating unsuitability of one or more of the keywords.

To compensate for mixed signals of the type just described, in an embodiment the training module 172 may look at that query usage in the context of the customer's overall session on the Web site. For example, a longer session, perhaps with multiple searches using one or more overlapping or common keywords, may indicate greater overall desirability of the common keyword(s) that the customer has employed for the queries run during the session. Individual queries using overlapping sets of keywords could be compared with each other to assess suitability of one or more of the overlapping or common keywords.

One way of looking at each session is as a vote of confidence, to a greater or lesser degree, in how a customer sees Web site contents as matching their query. Thus, for example, when a customer pauses to look at individual items in the list of query results, this may be considered to be a positive sign. The more such items a customer views, the more positive the user engagement perception may be. The longer a customer spends viewing one or more of the items, the more positive the user engagement perception may be.

Beyond spending significant time looking at one or more results of a particular query, the customer may do other things that indicate positive user engagement perception. For example, in an online marketplace in which the customer may be searching for a particular item that shows up in a set of query results, the customer may take the step of engaging with the seller. The customer may ask questions about the product, about its shipping, about seller return policies, or the like. The customer may put the particular query result, regarding the product, on a "wish" list or a "watch" list. In a "wish" list, as is known to ordinarily skilled artisans, a customer can put goods which s/he may be interested in acquiring at some time in the future. Prices for the goods could vary over time. An online marketplace could use the contents of the "wish" list to inform the customer of price changes. A "watch" list may function similarly. Additionally or alternatively, in an online marketplace in which customers bid on goods (as in an online auction), the online marketplace could use the contents of the "watch" list to inform the customer about time remaining in a particular auction. The online marketplace also could inform the customer about auctions involving identical or similar goods.

Different users may have different interactions with a Web site, even when searching for the same item. For example, consider a situation in which two users are searching for a particular brand, model, color, and configuration of smartphone. One user may enter a general query for the smartphone, for example, just the name and model, and may come up with hundreds or thousands of search results. In an online marketplace, for example, there may dozens or hundreds of sellers who post listings for a particular smartphone model, or a single seller may post multiple listings, one for each of the smartphone model the seller has available. The Web site may provide filter(s) for the user to narrow his/her search, for example, by color and/or configuration, by price, and perhaps by cellular telephone carrier. If the user has significant interest in purchasing the smartphone, the user may view one or more of the results of the query, either as originally presented or as filtered. If the original query turns up a very large number of results (impressions), and the user clicks on a few of those, it may be considered that that query has a relatively low click-through rate.

Each instance of filtering may be considered to be a different query, or may be considered to be part of the same original query. If the filtering narrows the query results (impressions), and the user clicks on a few of those, the click-through rate will be higher than if the user had clicked on the same results from the original query.

If one or more of the query results is of sufficient interest, the user may pose a question to the seller. The interaction with the seller may prompt the user to bid on the item (if the online marketplace includes an auction functionality and the seller is using that functionality to sell products); watch the item (to see how bidding proceeds on the item); or list the item, for example as an indication of an item that the user would like to purchase at some point. The user may decide to put the item in an online cart for purchase, or simply to buy the item outright (for example, if the online marketplace does not have auction functionality, or if the auction functionality includes an option to forego the bidding process and pay a predetermined price for the item).

Alternatively, that same user may decide that his/her original query was too broad, and may perform a different, narrower query. The narrower query likely would share at least one keyword in common with the original query, so that the queries might usefully be grouped together and considered to be part of the same session. Or, the user may decide that the smartphone for which s/he was searching is unsuitable, or simply that it would be desirable to search for a different smartphone. A query for a different smartphone may well use different keyword(s), and so may be considered to be a different session.

The other user may enter more detailed information for the same brand and model of smartphone, because the user may have a specific color, configuration, and carrier in mind. That user's query will produce fewer results that may be more relevant to that user. The user may follow the same practice as the first user, and view one or more of the query results, interact with seller(s), bid on the item, watch the item, put the item on a list, put the item in a cart, buy the item outright, etc.

If a user decides to buy the smartphone, then there is a purchase decision and revenue information derived from the purchase.

Multiple users may conduct a search for the same general brand and version of smartphone at the same time, or close together in time, or within a particular period of time. It may be that, in the case of a smartphone for example, such searches would be more prevalent within a certain period of time after introduction of a particular smartphone model and version. Fewer searches at later time periods may indicate lower popularity of the smartphone, or decreased user interest of some kind. Looking at all the data in aggregate from all these user searches, conducted within a suitable period of time, could inform as to suitability of particular keywords.

The extent to which users in this aggregate group actually buy a smartphone would indicate revenue value, which in turn may be compared with keyword bidding information to determine not only a keyword bidding budget for a particular category of item, but also how much to spend on keywords for particular item(s) within the category. Taken by itself, user purchase data may be too sparse to be statistically meaningful. However, taken in conjunction with the other data regarding user activity, the revenue data may be useful. Among the revenue data would be the price of an item. Of course, higher-priced items can generate the same revenue, for fewer sales, than would lower-priced items. Comparing the quantity of an item sold to the number user queries for the item, or user review of query results, also may inform as to the value of keywords on which to bid.

The foregoing discussion has focused on paid search, as for example when a user does a search using search engine 160, and an ad for the company's Web site shows up in the search results. However, ordinarily skilled artisans will appreciate that the technique(s) and structure described herein can have utility on any site that permits or publishes advertisements on which a user can click. Social media sites such as Facebook®, Twitter®, Google+®, and Pinterest® would be examples. That is, instead of bidding on keywords on a search engine, a user could bid on a social media site to have advertisements for goods corresponding to keywords placed on the social media site. The results of keyword selection would be just as applicable.

FIG. 2 shows a flow diagram of an embodiment for a keyword bidding process 200.

At 201, aggregated keyword server(s) 150 may monitor user sessions on the Website, at platform 102. At this point, a session may comprise a query, or it may comprise a direct click on an item that the user sees upon arriving at the Web site.

At 202, aggregated keyword server(s) 150 may monitor user queries on the Web site. As discussed earlier, depending on the nature of the queries, a user session may comprise multiple queries, or may comprise a single query.

After the user conducts a query, at 203 the server(s) 150 may monitor how the user interacts with the query results. Such interaction has been discussed in some detail above, and also will be discussed in detail below. For purposes of this discussion of FIG. 2, examples of user interaction or engagement may include interaction with a seller of an item; a bid on the item; a purchase of the item; a substantial amount of time spent reviewing results of a particular query; inputting of multiple queries on the same or similar item or category of items; etc.

At 204, aggregated keyword server(s) 150 combine or aggregate user engagement activity with respect to an item or category of items. Some of the activity may be favorable, and may point to the identification or selection of one or more keywords. Some of the activity may be less favorable or unfavorable, and may point away from the identification or selection of one or more keywords. By virtue of overall volume, the aggregate data should point in the correct direction.

In an embodiment, user engagement activity may be parsed in more detail, for example, to try to filter out clearly erroneous data (e.g. user(s) inputting widely disparate, unrelated queries during a single Web site visit), thus possibly skewing the data.

At 205, the keyword evaluation server(s) 170 compare the aggregated data for the item or category of items, with at least one criterion. In an embodiment, one particular criterion, such as impressions, may be the focus. If there is not a sufficient match, then flow may return to the beginning of FIG. 2. If there is a sufficient match between user engagement activity and least one predetermined criterion, then at 206 the server(s) 170 may parse the aggregate data for user engagement activity in more detail to identify potential keywords for the item or category of items.

At 207, the server(s) 170 may calculate a value score for each of the potential keywords identified in 206. In an embodiment, the value score may be related to a quantity or magnitude of one or more of the categories of user engagement activity. At 208, the keywords identified in 206 and having sufficiently similar value scores may be grouped. In an embodiment, such grouping may be limited to keywords with identical or very nearly identical value scores. The amount of variance of value scores around a particular value may be anywhere from 0-3%. The range or amount of variance is chosen to be as small as possible, while at the same time providing sufficient data to avoid the data scarcity problem discussed earlier. Thus, in certain circumstances there may be a relatively small number of keywords in a grouping, while in other circumstances there may be a relatively large number of keywords in a grouping. The number of keywords in the grouping should provide a sufficient amount of data, while avoiding, to the extent possible, a significant variance. In an embodiment, the value score may be determined based on a relationship to revenue per click (RPC). Other revenue or cost values may be used, including but not limited to RPI, CPC, or CPI, which were discussed above. A budget for keyword bidding also may be taken into account at this point in deriving the value score. In an embodiment, the budget may be taken into account at or just after 209, discussed below.

At 209, the value score of the group of keywords is compared to a cost of keyword bidding. In an embodiment, that cost is a cost of keyword bidding at the search engine 160, which as noted earlier may provide CPC, CPI, and other cost data. In an embodiment, if keyword bidding budget was not taken into account at 208, that budget may be taken into account at 209. Alternatively, if the group value score is greater than the cost of keyword bidding, there may be a subsequent step to see whether the bidding would fit into the keyword bidding budget.

At 210, if the group value score is greater than the cost of keyword bidding, a bid may be placed. As just noted, the decision to bid, notwithstanding the group value score, may be changed if the bid does not fit the budget.

FIG. 3 shows examples of user engagement activity 300. To some extent, these examples may overlap the examples provided earlier, but are discussed here in a less narrative format for organizational purposes. The following discussion is not intended to elevate one type of user engagement activity over another, so the ordering of the activity in this discussion is not critical.

One element of user engagement activity that may be relevant is the overall length of time a user spends on the Web site (session time spent 302). A user may access a Web site to start a session. During that session, the user may input one or more queries to search for an item or items. Alternatively, the user may browse an item or items appearing on one or more of the Web pages the user accesses at the site, without inputting a query. In general, the longer the user's session, the more relevant the Web site may be considered to be to the user's interests.

A session may be defined by a particular item or category of items for which a user searches, or the session may be defined by the user's overall time spent on the Web site. In the context of the techniques and structures described herein, identification of relevant keywords on which to bid can relate to a particular item or category of items. Accordingly, in an embodiment, to the extent that, during a particular Web site visit, the user searches for different items, each set of queries for a particular item or category of items may be considered to comprise a session.

Another possibly relevant element of user engagement activity relates to a number of impressions that a user's search query on the Web site produces (number of impressions 304). When a user inputs a query to search for a particular item or category of items, if there is a large number of search results containing the item or the category of items, the keyword(s) used in the query may be considered to be more relevant, and hence potentially more valuable.

A further possibly relevant element of user engagement activity would be click- through rate (CTR) 306. The number of times a user clicks on query results, as a percentage of the number of query results (impressions), can indicate a higher degree of relevance of the item or items that are the subject of the query, and thus may inform as to keyword(s) on which to bid.

As noted earlier, a user may conduct multiple queries while visiting the Web site

(number of searches 308). The queries may be for completely different products, or the queries may be for the same product or category of products. Where different products are the subject of different queries, each query or query set pertaining to a respective product may be considered a session, and relevance of keywords identified in each such session may be considered separately. Where the different queries, using different keywords, pertain to the same product or category of products, the user of different queries may indicate lower relevance of keywords in earlier queries during the session.

As a result of the queries that a user runs about an item, the user may make certain decisions short of purchasing or bidding on the item. For example, the user may decide to put a watch on the item (310). If the item on the Web site is available for sale for a limited period of time, the user may want to be made aware that available remaining time to purchase or bid on the item is decreasing. Short of watching a particular item for sale, a user may add an item or category of items to a product list (312) - a wish list, in one embodiment - indicating that the product is one in which the user has some current and potential future interest. Perhaps as a result of clicking through and reviewing one or more query results, a user may decide to add an item to an online shopping cart (314). Alternatively or additionally, the user may decide to ask the seller of the item a question about the item (316).

All of 310-316 indicate some kind of direct interest by a user in an item or category of items, and so would be relevant examples of user engagement activity which might be taken into account in devising one or more keywords on which to bid.

There may be circumstances in which a user puts forth a query, and some longer-than- usual period of time may elapse before the user does anything with the results of the query (e.g. click through, item watching, item listing, adding item to cart, asking the item seller a question, by way of non-limiting examples). Such longer period of time (318) may indicate that the user is considering the query results more carefully, and so may indicate potential higher relevance of keyword(s) in the user's query, and a corresponding potential desirability of bidding on the keyword(s).

Query results may span multiple pages. The number of pages that a user reviews from a particular query result (320) may correlate to the user's level of interest in the item(s) to which the query is directed. The larger the number of pages reviewed, the higher the potential relevance of the query term(s) as keyword(s) on which to bid.

In an online marketplace with an auction feature, a user may be sufficiently interested in an item that turns up in query results to bid on the item (322). The user may even decide to buy the item (324), either because the online marketplace does not have an online auction feature, or because the online marketplace allows a user to bypass the bidding process by agreeing to pay a certain amount for the item.

As a result of a user buying one or more items during a particular Web site visit, the online marketplace will generate revenue. The amount of revenue generated from a particular purchased item also may indicate desirability of bidding on the keyword(s) in a user's query.

It should be noted that a user may, and often will perform more than one of the actions indicated in the foregoing discussion of FIG. 3. By way of non-limiting example, the user may perform a query for an item; may click through on one or more of the results; decide to put an item on a watch list; ask the seller about the item; put the item in a cart; bid on the item; and/or buy the item.

In some of the preceding discussion, user engagement activity was compared to a predetermined criterion, in particular, a number of impressions (304 in FIG. 3). One or more of the other criteria may be used instead of, or in combination with the number of impressions. For example, session time spent (302 in FIG. 3); click-through rate (306 in FIG. 3); or item(s) watched (310 in FIG. 3) may constitute such criteria. This listing and the preceding discussion are intended to be exemplary, and are not intended to be exhaustive.

Looking back at the keyword evaluation server(s) 170 of FIG. 1 and the calculation of a value score in FIG. 2, it has been noted that the weighting of particular instances of user engagement activity can be a function of the volume of those instances of activity. How the algorithm ultimately assigns weights to the various categories of user engagement activity is not critical to the techniques and structure described herein. Depending on products or product categories, or product prices or product volumes, as well as analysis of user online browsing, shopping, and buying habits, certain types of user engagement activity may be more relevant to keyword value determination in some instances, and less relevant in others. For example, a certain product type may have a conversion rate (i.e. if someone goes online to look for that product, that person may be more likely to actually buy the product). Conversion rate for that product type, then, may be assigned a relatively large weight.

Certain embodiments are described herein as including logic or a number of components, modules, or mechanisms. Modules may constitute either software modules (e.g., code embodied (1) on a non-transitory machine-readable medium or (2) in a transmission signal) or hardware-implemented modules. A hardware-implemented module is a tangible unit capable of performing certain operations and may be configured or arranged in a certain manner. In example embodiments, one or more computer systems (e.g., a standalone, client, or server computer system) or one or more processors may be configured by software (e.g., an application or application portion) as a hardware-implemented module that operates to perform certain operations as described herein.

In various embodiments, a hardware-implemented module may be implemented mechanically or electronically. For example, a hardware-implemented module may comprise dedicated circuitry or logic that is permanently configured (e.g., as a special -purpose processor, such as a field programmable gate array (FPGA) or an application-specific integrated circuit (ASIC)) to perform certain operations. A hardware-implemented module may also comprise programmable logic or circuitry (e.g., as encompassed within a general- purpose processor or other programmable processor) that is temporarily configured by software to perform certain operations. It will be appreciated that the decision to implement a hardware-implemented module mechanically, in dedicated and permanently configured circuitry, or in temporarily configured circuitry (e.g., configured by software) may be driven by cost and time considerations.

Accordingly, the term "hardware-implemented module" should be understood to encompass a tangible entity, be that an entity that is physically constructed, permanently configured (e.g., hardwired), or temporarily or transitorily configured (e.g., programmed) to operate in a certain manner and/or to perform certain operations described herein. Considering embodiments in which hardware-implemented modules are temporarily configured (e.g., programmed), each of the hardware-implemented modules need not be configured or instantiated at any one instance in time. For example, where the hardware- implemented modules comprise a general-purpose processor configured using software, the general-purpose processor may be configured as respectively different hardware- implemented modules at different times. Software may, accordingly, configure a processor, for example, to constitute a particular hardware-implemented module at one instance of time and to constitute a different hardware-implemented module at a different instance of time.

Hardware-implemented modules can provide information to, and receive information from, other hardware-implemented modules. Accordingly, the described hardware- implemented modules may be regarded as being communicatively coupled. Where multiples of such hardware-implemented modules exist contemporaneously, communications may be achieved through signal transmission (e.g., over appropriate circuits and buses that connect the hardware-implemented modules). In embodiments in which multiple hardware- implemented modules are configured or instantiated at different times, communications between such hardware-implemented modules may be achieved, for example, through the storage and retrieval of information in memory structures to which the multiple hardware- implemented modules have access. For example, one hardware-implemented module may perform an operation, and store the output of that operation in a memory device to which it is communicatively coupled. A further hardware-implemented module may then, at a later time, access the memory device to retrieve and process the stored output. Hardware-implemented modules may also initiate communications with input or output devices and can operate on a resource (e.g., a collection of information).

The various operations of the exemplary methods described herein may be performed, at least partially, by one or more processors that are temporarily configured (e.g., by software) or permanently configured to perform the relevant operations. Whether temporarily or permanently configured, such processors may constitute processor-implemented modules that operate to perform one or more operations or functions. In some embodiments, he modules referred to herein may be processor-implemented.

Similarly, the methods described herein may be at least partially processor- implemented. For example, at least some of the operations of a method may be performed by one or more processors or processor-implemented modules. The performance of certain of the operations may be distributed among the one or more processors, not only residing within a single machine, but deployed across a number of machines. In some example embodiments, the processor or processors may be located in a single location (e.g., within a home environment, an office environment, or as a server farm), while in other embodiments the processors may be distributed across a number of locations.

The one or more processors may also operate to support performance of the relevant operations in a "cloud computing" environment or as a "software as a service" (SaaS). For example, at least some of the operations may be performed by a group of computers (as examples of machines including processors), with these operations being accessible via the network 104 (e.g., the Internet - see FIG. 1) and via one or more appropriate interfaces (e.g., APIs).

Embodiments may be implemented in digital electronic circuitry, or in computer hardware, firmware, software, or in combinations thereof. Embodiments may be implemented using a non-transitory computer-readable storage medium, (e.g., a computer program tangibly embodied in an information carrier, e.g., in a non-transitory machine-readable medium for execution by, or to control the operation of, data processing apparatus, e.g., a programmable processor, a computer, or multiple computers).

A computer program can be written in any form of programming language, including compiled or interpreted languages, and can be deployed in any form, including as a standalone program or as a module, subroutine, or other unit suitable for use in a computing environment. A computer program can be deployed to be executed on one computer or on multiple computers at one site or distributed across multiple sites and interconnected by the communication network 104.

In embodiments, operations may be performed by one or more programmable processors executing a computer program to perform functions by operating on input data and generating output. Method operations can also be performed by, and apparatus of example embodiments may be implemented as, special purpose logic circuitry, e.g., a FPGA or an ASIC.

The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network such as network 104. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. In embodiments deploying a programmable computing system, it will be appreciated that both hardware and software architectures merit consideration. Specifically, it will be appreciated that the choice of whether to implement certain functionality in permanently configured hardware (e.g., an ASIC), in temporarily configured hardware (e.g., a combination of software and a programmable processor), or a combination of permanently and temporarily configured hardware, may be a design choice. Hardware (e.g., machine) and software architectures that may be deployed in various embodiments now will be described.

FIG. 4 is a diagrammatic representation of an exemplary computer system 400, within which a set of instructions may be executed to cause the machine to perform any one or more of the methodologies discussed herein. The computer system 400, in whole or in part, may be embodied in any of the servers or other machines shown in FIG. 1.

In some embodiments, the machine operates as a standalone device, or may be connected (e.g., networked) to other machines. In a networked deployment, the machine may operate as a server or a client machine 110 or 112 in a client-server network environment, or as a peer machine in a peer-to-peer (or distributed) network environment. The machine may be a personal computer (PC), a tablet, a set-top box (STB), a personal digital assistant (PDA), a smartphone, a web appliance, a network router, switch or bridge, or any machine capable of executing a set of instructions 424 (sequential or otherwise) that specify actions to be taken by that machine. Further, while only a single machine is illustrated, the term "machine" shall also be taken to include any collection of machines that individually or jointly execute a set (or multiple sets) of instructions 424 to perform any one or more of the methodologies discussed herein.

In an embodiment, the computer system 400 may include a processor 402 (e.g., a central processing unit (CPU), a graphics processing unit (GPU, or both)), a main memory 404 and a static memory 406, which communicate with each other via a bus 408. The computer system 400 may further include a video display unit 410 (e.g., a liquid crystal display (LCD), a plasma display, a light-emitting diode (LED) display, or a cathode ray tube (CRT) display). The computer system 400 also may include an alphanumeric input device 412 (e.g., a keyboard or keypad), a UI navigation device 414 (e.g., a mouse or touchpad), a disk drive or other storage (e.g. solid-state device, or SSD) unit 416, a signal generation device 418 (e.g., a speaker), and a network interface device 420.

The disk drive unit 416 may include a machine-readable medium 422 on which is stored one or more sets of instructions and data structures (e.g., instructions 424) embodying or utilized by any one or more of the methodologies or functions described herein. The instructions 424 may also reside, completely or at least partially, within the main memory 404 and/or within the processor 402 during execution thereof by the computer system 400, with the main memory 404 and the processor 402 also constituting machine-readable media 422.

The instructions 424 may further be transmitted or received over a network 426

(which is not part of the computer system 400) via the network interface device 420, utilizing any one of a number of well-known transfer protocols (e.g., HTTP).

While the machine-readable medium 422 is shown in an embodiment to be a single medium, the term "machine-readable medium" should be taken to include a single medium or multiple media (e.g., a centralized or distributed database, and/or associated caches and servers) that store the one or more sets of instructions 424. The term "machine-readable medium" shall also be taken to include any medium that is capable of storing, encoding, or carrying a set of instructions 424 for execution by the machine and that cause the machine to perform any one or more of the methodologies of the present disclosure, or that is capable of storing, encoding or carrying data structures utilized by or associated with such a set of instructions 424. The term "machine-readable medium" shall, accordingly, be taken to include, but not be limited to, solid-state memories, optical media, and magnetic media.

As alluded to earlier, particular sequences of operations in various embodiments are provided for convenience, and are not intended to imply that the same order is required in all embodiments. Any operational sequence in which a subsequent operation does not depend on a previous one is appropriately within the scope of this disclosure. By way of nonlimiting example, there has been discussion of various types of user engagement activity with reference to FIG. 3. The order of discussion of those activity types does not imply a sequence of consideration, nor does the discussion order imply an order of importance.

The Abstract of the Disclosure is provided to comply with 37 C.F.R. § 1.72(b), requiring an abstract that will allow the reader to quickly ascertain the nature of the technical disclosure. It is submitted with the understanding that it will not be used to interpret or limit the scope or meaning of the claims. In addition, in the foregoing Detailed Description, it can be seen that, in some cases, various features may be grouped together to streamline the disclosure. This method of disclosure is not to be interpreted as reflecting an intention that the claimed embodiments require more features than are expressly recited in each claim. Rather, the following claims reflect subject matter including fewer than all features of a single disclosed embodiment. Thus, the following claims are hereby incorporated into the Detailed Description, with each claim standing on its own as a separate embodiment.

Although the present disclosure has been described with reference to certain embodiments, it will be evident that various modifications and changes may be made to these embodiments without departing from the broader spirit and scope of the disclosure. Accordingly, the specification and drawings are to be regarded in an illustrative rather than a restrictive sense.