[0001 ] The present application claims priority from U.S. provisional patent application no. 62/165,529, filed May 22, 2015 and entitled "SYSTEM AND METHOD FOR RANKING SEARCH RESULTS", the disclosure of which is hereby incorporated by reference in its entirety.

Background:

[0002] Optimizing search results relevancy is highly challenging in an information retrieval system. First, the user's query must be matched with many potential search results, which is complex when queries are too general (i.e. matching too many documents), contain misspelled words or simply use the wrong vocabulary (i.e. words not present in the index). In addition, in order to optimize relevancy, it is not enough to only use the user's query, because it is often not expressive enough to understand what the user needs. Moreover, in most cases, the environment is constantly changing. For example, content is added or modified, new users are introduced to the system or existing user's needs evolve. Thus, results considered relevant at a given moment may not be the most relevant ones, days, weeks or months later. Therefore, the determination of the relevance of search results must evolve over time to reflect such contextual changes.

[0003] Hence, there is a need for an improved system which, by virtue of its design and components, would be able to overcome some of the above- discussed prior art concerns.

Summary:

An object of the present is to provide a method and system which, by virtue of its design and components, satisfies some of the above-mentioned needs and is thus an improvement over other related ranking systems and/or methods known in the prior art. In accordance with an aspect, there is provided a method for ranking search results, the method comprising: tracking a plurality of search sessions, each search session comprising one or more search queries made to a search index and one or more accessed search results, each search query defining at least one query element and generating a plurality of search results, and each accessed search result corresponding to one of the generated search results which is accessed by a user during the search session; for each accessed search result occurring across the plurality of search sessions, associating the accessed search result with each search query executed within a given search session, prior to the user accessing the accessed search result; calculating for a preselection of query elements to be indexed, a result score for each pairing of a query element from the preselection and an associated accessed search result; and generating a ranking model containing the result score in association with the corresponding pairing, for ranking subsequent search results from subsequent search queries according to the result scores.

In accordance with another aspect, there is provided non-transitional storage comprising data and instructions for execution by a processor, to carry out the steps of the above-mentioned method.

In accordance with yet another aspect, there is provided a system for ranking search results, the system comprising: a User Interface Module, embedded in a processor, for tracking a plurality of search sessions, each search session comprising one or more search queries made to a search index and one or more accessed search results, each search query defining at least one query element and generating a plurality of search results, and each accessed search result corresponding to one of the generated search results which is accessed by a user during the search session; an Events Database for associating each accessed search result occurring across the plurality of search sessions, with each search query executed within a given search session, prior to the user accessing the accessed search result; an Events Analyser, embedded in the processor, for calculating, for a preselection of query elements to be indexed, a result score for each pairing of a query element from the preselection and an associated accessed search result; and a Query-to-Search-Result Model Database for storing an ranking model containing the result score in association with the corresponding pairing, for ranking subsequent search results from subsequent search queries according to the result scores.

Other objects, advantages and features of the present invention will become more apparent upon reading of the following non-restrictive description of preferred embodiments thereof, given for the purpose of exemplification only, with reference to the accompanying drawings.

Brief description of the drawings:

[0004] FIG. 1 is a block diagram showing components of a system, according to an embodiment.

[0005] FIG. 2 is a schematic representation of a user session, according to an embodiment.

[0006] FIG. 3 is a flow chart showing steps of a method of building a ranking model for a search engine, in accordance with an embodiment.

[0007] FIG. 4 is a flow chart showing steps of a method of ranking search results, in accordance with an embodiment.

Detailed description of preferred embodiments:

[0008] In the following description, the same numerical references refer to similar elements. The embodiments mentioned and/or geometrical configurations and dimensions shown in the figures or described in the present description are embodiments of the present invention only, given for exemplification purposes only.

[0009] One or more systems described herein may be implemented in computer programs executing on programmable computers, each comprising at least one processor, a data storage system (including volatile and non-volatile memory and/or storage elements), at least one input device, and at least one output device. For example, and without limitation, the programmable computer may be a programmable logic unit, a mainframe computer, server, and personal computer, cloud based program or system, laptop, cellular telephone, smartphone, tablet device, or video game console.

[0010] Each program is preferably implemented in a high level procedural or object oriented programming and/or scripting language to communicate with a computer system. However, the programs can be implemented in assembly or machine language, if desired. In any case, the language may be a compiled or interpreted language. Each such computer program is preferably stored on a storage media or a device readable by a general or special purpose programmable computer for configuring and operating the computer when the storage media or device is read by the computer to perform the procedures described herein. In some embodiments, the systems may be embedded within an operating system running on the programmable computer. In other example embodiments, the system may be implemented in hardware, such as within a video card.

[001 1 ] Furthermore, the systems, processes and methods of the described embodiments are capable of being distributed in a computer program product comprising a computer readable medium that bears computer-usable instructions for one or more processors. The medium may be provided in various forms including one or more diskettes, compact disks, tapes, chips, wireline transmissions, satellite transmissions, internet transmission or downloadings, magnetic and electronic storage media, digital and analog signals, and the like. The computer-usable instructions may also be in various forms including compiled and non-compiled code.

[0012] Broadly described, as exemplified in the accompanying drawings, there is provided a novel system for improving search results relevancy based on the user's interactions with a result ranking system 100. User actions are accounted for implicitly rating of search results, allowing the system to better identify the most relevant search results for a subsequent search query. This allows to shorten the user's search session by predicting the better search results, even if for example, the query is ambiguous or misspelled, by using past users' query rewriting, search results accessed, other interactions, and/or the like. In one embodiment, the system defines positive and negative outcome custom events of a user session, thereby allowing the system to learn to better identify the most relevant search results, in order to avoid negative outcomes and promote positive outcomes. Moreover, the number of steps required by the user to reach the information he/she is looking for is reduced.

[0013] An information retrieval system is used by a user to retrieve information being sought after from a collection of content. In a computer- implemented information retrieval system, such as a search engine, a user enters a search query via a user interface and the information retrieval system retrieves search results that are considered relevant to the search query. The search results each link to a piece of content stored in the collection of content. The relevancy may be based on a match of a portion of the search query with content from the collection of content. The content of the collection may be indexed. The search result generated by the information retrieval system may be presented within the user interface and the user may access (ex: by interacting, such as a mouse click) one or more of the search result to further access the content linked to the search result.

[0014] It was observed that an individual search query made by a user is not the only piece of information that indicates what content the user is seeking. Multiple search queries made by the user within a short time period may provide additional information as to what content the user is seeking. Similarly, the user's interactions with the information retrieval system, such as accessing of search results, navigation of the information retrieval system and interaction with specific pieces of content may provide further information about what content the user is seeking.

[0015] An object of the present is to leverage such user interactions (for example, through positive or negative interpretation of the interactions) in order to improve interpretation of a user's query and to the relevance of the search results presented in response to search queries made by the user. [0016] Referring now to Figure 1 , therein illustrated is a schematic diagram of a result ranking system 100 according to one example embodiment. The system includes a User Interface module 102, a Query Pipeline module 104, a Search Index module 106, a Top Search Results Service module 108, an Events Database 1 10, an Events Analyzer module 1 12, and a Query to Search Result Model database 1 14. It will be understood that the result ranking system 100 may correspond to a fully functional information retrieval system for identifying content relevant to user queries. The User Interface module 102, Query Pipeline module 104, Search Index module 106, Top Search Results Service module 108 and Events Analyzer module 1 12 may be implemented in a processor of the computing device. The Events Database and Query to Search Result Model database 1 14 may be implemented in a storage device.

[0017] In a model building mode of the result ranking system 100, the user Interface module 102 captures user action events within the result ranking system 100. Such user action events include standard search events (e.g. executing a search query, refining by metadata, accessing a search result) and custom events specific to a particular system (e.g. products added to a shopping cart, support case created, etc.). These user action events are tracked and recorded in the Events Database 104 and are made available to the Events Analyzer module 1 12, which in turn uses these events to build ranking models which are stored in the Query to Search Result Model database 1 14. A ranking model provides an indication of search results accessed by a user in a plurality of search sessions, as described elsewhere herein.

[0018] In a searching mode of the result ranking system 100, when a subsequent search query is made by a user, the search query is sent via the User Interface module 102 to the Query Pipeline module 104. The Query Pipeline module 104 further calls the Top Search Results Service module 108 with the search query and receives therefrom a pre-ranked list of search results based on the ranking model of the Query to Search Result Model database 1 14. The Query Pipeline module 104 then uses the pre-ranked list of search results and creates ranking functions for these search results. The Query Pipeline module 104 sends the query with the ranking expressions to the Search Index module 106. In response to receiving the search query, the Search Index module 106 generates a second list of search results. The Search Index module 106 may carry out functions of a typical search engine to identify search results relevant to the search query that are included in the second list of search results. The search results of the second list are combined with the pre-ranked list of search results. The combined list of search results are then returned to the User Interface module 102 and presented to the user. The pre-ranked list of search results from the Query to Search Result Model database 1 14 influence the search results that are returned to the User Interface module 102 and that are presented as search results in response to the search query. The ranking of the search results from the pre-ranked list and the second list of search results are determined based on the ranking functions for the pre-ranked list. The ranking functions may serve to promote the ranking of results in the pre-ranked list.

[0019] It will be understood that the result ranking system 100 may operate simultaneously in the model building mode and the searching mode. For example, as a user is carrying out user action events via the user interface 102 to identify content being sought, the result ranking system 100 is operating in the searching mode in that search queries made by the user generate the combined list of search results from the Search Index module 106 based on existing ranking models stored within the Top Search Results Service module 108. At the result ranking system 100 is also operating in the model building mode in that these user actions events are tracked and recorded in the Events Database 100 and may be used for updating ranking models stored in the Query to Search Result Model database 1 14.

[0020] Continuing with Figure 1 , the User Interface module 102 tracks a plurality of search sessions. A search session includes a plurality of user action events of a user made via the user Interface module 102. The user action events of a search session includes one or more search queries entered into the User Interface module 102 and made to the Search Index module 106. Each search query defines at least one query element and in response to a search query the Search Index module 106 may generate one or more search results. In some cases, the Search Index module 106 may generate zero search results in response to a search query if no relevant search results are found.

[0021 ] The at least one query element defined by a search query corresponds to an element which provides an indication of information being sought by the user making the search query. For example, the search query may include a string of one or more words. A query element of a search query may include the complete search query (i.e. the entire string of one or more words), components of the search query, a derivative of the search query, a component of such a derivative of the query, and/or the like. For example, a query element may be a full query, a stemmed query, a full stemmed query, a word of a query, a tokenized original query, a stemmed word of a query, an original word of a query, and/or any combination thereof.

[0022] The search results correspond to content elements from the collection of content elements being searched that has some relevance to at least one query element of the search query as determined by the Search Index module 106. Each search result may link to a content element within the collection of content, such as a file or document (text, image, video, etc.), a webpage, a location on a webpage or other document, which is accessible via a communication network, for example the Internet, a private or local network, and/or the like. A search result may also refer to said hyperlink associated to the content element. In the present, an "accessed search result" (or "clicked search result") refers to a search result having been clicked, selected, viewed, accessed, and/or involving any other interaction by the user, and which represents a level of relevance of the search result to this user. A search result generated from a search query and that has been accessed by a user during a search session is herein referred to as an "accessed search result".

[0023] The beginning of the search session may be defined the user beginning a series of user action events via the User Interface module 102. For example, the beginning of the search session may correspond to a user logging into a portal, such as a user portal presented on web browser, desktop application, or mobile application. The end of the search session may be defined by the end of the series of user action events, such as the user logging out, timing out, or closing the portal. Within the search session in which the user accessed one or more search results, the search result that was last accessed by the user prior to the end of the search session is herein referred to as a "last accessed search result".

[0024] A user action event may also be a "custom event". A custom event herein refers to a user action event other than a search query or accessing a search result but that is indicative of either a positive outcome or negative outcome regarding the user finding the content being sought. For example, a positive outcome custom event may include a product being added to a shopping cart on an e-commerce site. For example, a negative outcome custom event may include a user creating a support case in a "self-service" website for customer service.

[0025] It was observed that where a plurality of search queries are made within a search session, successive search queries made may be representative of refinements of query elements included in the search queries that lead to a search result being sought by the user.

[0026] It was also observed that the last accessed search result of a search session may link to the content element being sought by the user.

[0027] It was further observed that certain custom events may also be indicative of finding a result being sought or not finding that result.

[0028] Although these hypotheses are not always representative of reality in every situation, they are considered to accurately represent the normal behavior of users over a large number of search sessions.

[0029] A search session is represented by all the user action events of a user with the result ranking system 100 through the user interface 102 in that search session. FIG. 2 schematically illustrates a user session 200 according to one example embodiment. During a user session 200, a user may execute multiple search queries 202, 212 and may or may not access search results generated from these search queries 202, 212. In the illustrated example, of the search results generated from the first search query 202, search results 204, 206 were accessed. Break 208 is representative of other user action events made by the user within the search session. A custom event 210 was further made. Furthermore, of the search results generated from the second search query 212, search results 214, 216, and 218 were accessed. The accessed search result 218 was accessed after 214 and 216 and represent the last clicked search result 218. It will be understood that the order of the occurrences of user action events within a search session are tracked. These user action events and their relative order are recorded within the Events Database 1 10.

[0030] FIG. 3 is a schematic diagram illustrating the operational steps executed by the Events Analyzer module 1 12 to build the Query to Search Result model 1 14 (also herein referred to as a "ranking model") from the information in the Events Database 1 10. The operational steps of the Event Analyzer module 1 12 may correspond to its submodules. The Query to Search Result model is built based on the aforementioned observation that a user refines his or her queries over the course of a search session to reach the content element being sought.

[0031 ] Continuing with Figure 3, the Events Analyzer module performs a method 300 of building a ranking model, including at step 310 (a) collecting query elements from multiple search sessions carried out via the search engine, and for each search session, storing each query element in association with an accessed search result of the corresponding search session; at step 312, (b) determining a preselection of query elements to be associated to the accessed search result, representing the most frequent query elements leading to the associated accessed search result; at step 314, (c) calculating a score for each pairing of accessed search result and query element in the preselection of step (b); and at step 316, (d) generating a model containing the calculated score in association with the corresponding pairing, in order to rank the accessed search results according to the score when a given subsequent query is executed.

[0032] More particularly, at step 310, search sessions are filtered and loaded. For example, the set of user action events that form together a search session are identified from the user action events recorded in the Events Database 1 10. A search session may be associated to all search queries, accessed search results and custom events carried out by the user during the search session. The search session is further defined by its duration. A search session begins with the user accessing the user interface module 102 and ends after a period of inactivity or the user logging out. The last accessed search result of each search session may also be identified.

[0033] In one example embodiment, sessions comprising at least one negative outcome custom event are filtered out. An example of a negative outcome may involve the system logging a case creation event when the user creates a support case, which shows that the current session was unsuccessful. Other examples of sessions with negative outcomes include sessions without any accessed search results or sessions without at least one non empty query. Search sessions that are filtered out are not taken into account for building the ranking model.

[0034] In another embodiment, sessions with at least one negative outcome custom events are not ignored, but search queries and search results of these sessions may be demoted within the ranking model.

[0035] In another example embodiment, search sessions with positive outcome custom events are promoted within the ranking model to reinforce the link between the session queries and the accessed search results.

[0036] At step 312, search queries are logically associated to search results across a plurality of search sessions. For example, the associating of search queries to search results may be carried out for each of the search sessions to be used for building the ranking model. [0037] More particularly, for each accessed search result occurring across the plurality of search sessions, the accessed search result is associated with each search query executed within the search session of the accessed search result prior to the user accessing the accessed search result. Referring back to Figure 2, accessed search result 204 is associated to search query 202 that preceded it. Similarly, accessed search result 206 is also associated to search query 202. Accessed search result 218 is associated to both search query 202 and search query 212 since both of these preceded the accessed search result 218 being accessed within the search session 200. Similarly, each of accessed search results 214 and 216 are also associated to search queries 202 and 212.

[0038] In one example embodiment, a preselection of query elements to be mapped in the ranking model is determined.

[0039] In another example embodiment, the preselection of query elements to be mapped may correspond to all of the query elements of all the query searches across all of the search sessions considered for building the rank model.

[0040] The associations of accessed search results to search queries leading up to the accessed search results are further analyzed to determine for each unique query element, the number of accessed search results that is associated to that query element. This count of number of associated accessed search results represents a query frequency score for the query element. That is, for each query element, a frequency score representing the number of times the query element appears in a search query associated to an accessed search result is detected.

[0041 ] In one example embodiment, the query frequency score may be a normalized count, wherein the normalizing is carried out in the range [a,b] = [1/sqrt(N), 1] using the following equation: where N is the search result clicked count, min is the minimum count and max is the maximum count.

[0042] For example, those query elements that have a query frequency score exceeding a predetermined threshold may be selected to the preselection of query elements to be mapped. These may correspond to the query elements that most frequently led to a search result being accessed.

[0043] In one example embodiment, the types of query element determined from a query search may a tokenized original query, a tokenized and stemmed query, and as a full stemmed query, and a frequency score is determined for each query elements by type. From the frequency scores for each query element, a corresponding set of lists containing query elements may be generated by type. More particularly, four query lists are generated, which correspond to the preselection of query elements to be mapped:

(1 ) OriginalWordList which contains a selection of tokenized original queries having the highest normalized count (second set).

(2) StemWordList which contains a selection of tokenized and stemmed queries having the highest normalized count (third set).

(3) StemQueryList which contains a selection of full stemmed queries having the highest normalized count (fourth set).

(4) StemWordAIIList which contains all tokenized and stemmed queries (a first set).

[0044] Continuing with Figure 3, at step 314, for the preselection of query elements to be indexed, an accessed result score for each pairing of a query element from the preselection and an associated accessed search result is calculated. That is, for each query element of the preselection, a pairing is formed with each accessed search result associated to the query element and a result score is calculated for that pairing.

[0045] According to one example embodiment, the result score may be determined based on the query frequency score of the query element of the pairing. It will be appreciated that the query frequency score of the query element indicates the number of times a query element was used that led to a search result being accessed, which provides a metric of the popularity of the query element.

[0046] According to one example embodiment, for a subset of the preselection of the query elements, the result score for each pairing of the subset is determined based on the number of search queries associated to the accessed search result of the pairing. The result score is further determined based on the average number of search queries associated to an accessed search result. The number of search queries associated to a given accessed search result indicates the number of search queries made within a search session prior to that search result being accessed by a user. The subset of the preselection may be the query elements in the first set, the second set and the third set.

[0047] According to one example embodiment for calculating the result score for a pairing of a given query element with a search result associated to it, an inverse search result frequency (IDF) is calculated for the given query element q, using the following equation:

where ndt is the total number of accessed search results and nd _q is the number of search results accessed for the query element (i.e. the number of search results that followed any search query that included the query element). Then, the result score (docScore) is calculated for each accessed search results associated to that query element. More particularly, a docScore is calculated for each combination of accessed search result and the associated query element using the following Okapi BM25 equation:

where nd _q is the number of search results accessed for the query element, K is a constant with a default value of 1 .2, B is a constant with a default value of 0.75, nq _d is the number of search queries that led to the search result and anq _d is the average number of search queries leading to a search result.

[0048] According to an alternate embodiment, the score calculation includes a multiplier representing an upgrade or downgrade of the search result (or promotion/boosting and demotion). More particularly, an upgrade multiplier is used when a query and search result pairing happened in a search session with a positive outcome custom. Conversely, a downgrade multiplier is factored in when a query / search result pairing happened in a session with a negative outcome custom event.

[0049] Alternatively or additionally, a time decay multiplier is factored into the result score calculation. The time decay multiplier takes into account an "age" of a query / search result pairing, so as to gradually decrease the weight of a particular search result as time lapses.

[0050] Alternatively or additionally, an information gain multiplier is factored into the score calculation. The information gain multiplier takes into account how useful a word is when included in a query element to find interesting documents.

[0051 ] At step 314, a ranking model is generated containing the result scores in association with corresponding pairing. The pairings of query elements and associated accessed search results are stored within the ranking model in associated with their result score. The ranking model may be used to rank accessed search results associated to query elements stored in the ranking model that match query elements of subsequent search queries made to the Query pipeline module 104. The ranking may be performed according to the result scores. The ranking model may be stored within the Query to Search Result model 1 14.

[0052] According to the example in which four query lists are generated as the preselection of query elements to be mapped, four maps may be generated for the ranking model and stored within the Query to Search Result model. The four maps are: (1 ) stemQueryMap which contains for each stemmed query, the most accessed search results with their search result score (i.e. normalised search result count calculated according to equation 1 ).

(2) stemWordAIIMap which contains for each stemmed query word, all accessed search results with their search result score as calculated using equation 3.

(3) originalWordMap which contains for each original query word, the most accessed search results with their search result score as calculated using equation 3.

(4) stemWordMap which contains for each stemmed query word, the most accessed search results with their search result score as calculated using equation 3.

[0053] It will be appreciated that search queries are accumulated during a session and associated with each accessed search result (i.e. "accessed search results"). After multiple sessions are processed, each accessed search result is associated with a list of query elements ranked based on their frequency. The most frequent query elements may be preselected. Then, a search result score is calculated for each pairing of preselected query element and associated search result. The ranking model is composed of a list of query elements, and for each one of them a list of search results and result scores. It will be appreciated that since the ranking model for an accessed search result includes all search queries done within a session before the search result was accessed, the model takes into consideration the manual query rewriting accomplished by users, such as mistyping or misspellings by the user.

[0054] The ranking model may be built on a per user basis, wherein one ranking model is built for each user. Accordingly, the Events Analyzer module 1 12 considers only search sessions made by that user when building a ranking model. The mappings of query elements to associated search results are customized for that user. [0055] Alternatively, the ranking model may be built for a group of users, wherein a single ranking model is built for the group of users. The group of users may be from the same organization or department and may have similar use cases. Accordingly, the Events Analyzer module 1 12 considers search sessions made all of the users within the group. The mappings of query elements to associated search results are customized for that user group.

[0056] The Events Analyzer 1 12 may periodically update or rebuild the ranking model. For example, after the User Interface module 102 tracks and records newer user action events into the Events Database 1 10 while the result ranking system 100 operates in the searching mode, the Events Analyzer module 1 12 may receive search sessions corresponding to the newer user action events and update or rebuild the ranking model based on the newer user action events.

[0057] Referring back to FIG. 1 , the Query to Search Result Model database 1 14 is then used by the Top Search Results Service module 108 to return a pre-ranked list of search results from the accessed search results mapped in the model that are associated to query elements that match subsequent search queries made to the Query Pipeline module 104. When a user enters a subsequent query, it is sent to a query pipeline module 104 that is responsible for processing the query pipeline module 104, calling the Search Index module 106 and processing the results before sending them back to the user interface 102. Prior sending the search query to the Search Index module 106, the Query Pipeline module 108 calls the Top Search Results Service with the subsequent search query and receives a pre-ranked list of at least one search result stored in the Query to Search Result Model 1 14 that best fit the subsequent search query.

[0058] Referring now to Figure 4, therein illustrated is a method 400 for identifying and ranking accessed search results of the ranking model that match subsequent search queries. The method 400 may be carried out by the Top Search Results Service module 108, the method including (a) providing a ranking model generated from the example methods described above, said ranking model comprising mappings of pairings of query elements with associated search results and result scores; at step 410 (b) receiving a subsequent search query, comprising one or more subsequent query elements; at step 414, (c) determining parameters of the subsequent search queries, such as the length of the subsequent search query; (d) from said ranking model, selecting accessed search results mapped in the ranking model in associated with a query element matching said one or more subsequent query elements of the subsequent search query (i.e. accessed search results stored in the Query to Search Result Model module 1 14); at step 416, (e) for each selected search result, calculating a ranking score based on the parameters determined at step (c); and (f) ranking the selected search results according to the score for outputting a pre-ranked list of said selected accessed search results at step 418. As a result, at least one accessed search result stored in the ranking is identified based on a match of at least one subsequent query element with a query element stored in the ranking model.

[0059] More particularly, at step 412, the subsequent query is cleaned by removing unwanted characters (e.g. punctuations), stemming words and removing stop words.

[0060] At step 414, parameters of weighting coefficients are selected. The parameters are used to differently weigh result scores associated to pairings of query elements and associated search results stored in the ranking model. The parameters may be selected based on the length of the query. The different parts of the ranking model have varying importance depending on the length of the query. Therefore, the selection of parameters may be made as a function of the length of the query. Different parameter sets are associated with different query lengths.

[0061 ] In the example embodiment wherein the ranking model comprises the four mappings stemQueryMap, stemWordAIIMap, originalWordMap and stemWordMap , each parameter set contains five parameters: four weighting coefficients (C _S QM, C _S WM, CSWAM, CQWM) and one filter component. The coefficients (CSQM, CSWM, CSWAM, COWM) are used to influence how the result scores are combined and the filter component (F _S QM) is used to choose how many search results will be kept from the stemQueryMap.

[0062] The sets of parameters for all query lengths may be learned using the data in the Events database 1 10 (FIG. 1 ) based on machine learning applied to the plurality of search sessions for prioritizing the last accessed search results of the plurality of search session. More particular, the sets of parameters are learned according to a parameter optimization process which splits the events in one training set and one test set. First, a model is learned on the training set, as specified previously. Then, different sets of parameters are tested on the test set. The objective is to maximize, for all search sessions in the test set, the percentage of times the last clicked search result of the session is returned in the top 5 results for all queries in the search session. At each iteration, during the parameter optimization process, the parameters tested are carefully chosen using an algorithm that mixes different optimization techniques (i.e. randomized weighted grid search, simulated annealing and genetic algorithms). This algorithm allows to efficiently explore the space of parameter values in order to rapidly estimate the global maximum. A simulated annealing approach is used in which the probability to select a pseudo random point in the parameter space decreases over time. At each iteration, a pseudo random point is selected or one neighbour of an existing point is selected. The pseudo random points are chosen in a way to maximize the distance between tested points. If not in the random condition, an existing point is selected using a weighted random process and one of its neighbours is selected. Possible neighbors are generated using a mutation process in which each parameter value is modified using a Gaussian random process. After a fixed number of iterations or after a number of iterations without improvements, the point (set of parameters) with the highest evaluation value is returned.

[0063] At the end of the parameter optimization process, the best parameter sets for all query lengths are saved in buckets to reduce the number of parameter sets to keep (e.g. one set for queries of length 1 word, one set for length 2 and 3 words, one set for length 4 to 8 words and one set for length greater than 8 words). This process allows identifying suitable parameter sets for the specific events recorded and for all possible query lengths, within a particular system 100 (FIG. 1 ). The length may be determined based on the number of words in the query (preferably after having been purged of unnecessary characters or certain words in the query), or based on any other suitable measure of elements in the query.

[0064] When a query is executed (at step 410), the following equation is used, in order to select the right set of parameters for the query length. The equation returns the index of the parameter set {pSet) based on the query length (<7s/ze) and the number of parameter sets (nSets).

pSet(q _size , nSets) = max 0, min ^nSets — 1, floor (^ _^(2) ^) ^ (4)

[0065] At step 416, a combined ranking score is calculated for the search results associated to entries in the ranking model that match one or more of the subsequent query elements. Firstly, stored search results of the ranking are selected from each map of the ranking model. Then a score is calculated for each selected accessed search result.

[0066] There are four subsets of search results to include in the result list based on the learned models, namely:

(1 ) search results from the stemQueryMap (SQM) associated with the exact same stemmed query (the maximum number of search results is specified by the filter parameter F _S QM);

(2) search results from stemWordAIIMap (SWAM) that have in their model the conjunction of all query words (stemmed words and without stop words);

(3) search results from stemWordMap (SWM) that have in their model the disjunction of all query words (stemmed words and without stop words); and (4) search results from originalWordMap (OWM) that have in their model the disjunction of all original query words (not stemmed and without stop words).

[0067] When all potential search results are selected from a matching of their query model with the current query, a combined ranking score is calculated, for each search result, using the following equation: score { ) — E _S Q _M C _S Q _M S _S Q _M + E _SWAM C _SWAM S _SWAM + E _SWM C _SWM S _SWM + E _0WM C _0WM S _0WM (5) where S _x is the search result score in the map x; C _x is the coefficient, in the parameter set selected (based on the previously identified index pSet), for the map x; and E _x equals 1 if the search result d was selected in the map x, 0 otherwise.

[0068] In one embodiment, for each query element, the search result score is further weighted with the information gain score of the query element, thereby giving more weight to query elements that normally lead to documents selected by users.

[0069] The result list is then ranked according to the calculated scores and the highest scoring search results, are then returned to the Query Pipeline together with their scores within a pre-ranked list of accessed search results, step 418. The scores may act as ranking functions for ranking the selected accessed search results when combined with search results identified by the Search Index module 106.

[0070] In the query pipeline 104, ranking functions are added to the subsequent search query for these search results to inform the index server 106 to increase their ranking score. The subsequent search query with the new ranking functions is then sent to the search index 106. The search index 106 generates subsequent search results for the subsequent query according to known search methods and further includes the ranked list of previously accessed search results received from the Query Pipeline 104. The inclusion of the ranked list of accessed search results and their ranking with respect to the subsequent search results generated by the search index 106 are determined based on the ranking functions. A combined list of results that includes the subsequent search results and the ranked previously accessed search results are then sent back to the user interface module 102. This combined list may be presented, such as being outputted, for example on a display screen (or any other suitable output device).

[0071 ] Advantageously, the system provides an automatic ranking improvement process within complex search infrastructures which may include: heterogeneous information sources, different use cases, many user roles, evolving content, and/or the like. This self-improving system advantageously takes into consideration users interactions with the system to improve the search experience by helping users find useful information faster, reducing the necessity for users to over specify their query or to learn the exact vocabulary used within the database being queried. In order to accomplish this, the system uses data gathered by an event logging system that records the main actions accomplished by users in the user interface. The events are used to build a model linking queries to search results, which is then used at query time to influence the list of search results displayed to the user and their ranking.

[0072] Advantageously, embodiments of the system described herein optimize search result relevance, automatically, based on the user interactions. The system records user actions in the user interface 102 and uses this information to learn about the relationships between queries used by the users and the desired search results, i.e. search results producing positive outcomes as defined by the owner of the system. Moreover, by automatically injecting ranking functions into a subsequent query, based on the learned ranking model, the system can continuously learn from the users' interactions and improve the search experience.

[0073] Although the preferred embodiment as illustrated in the accompanying drawings comprises components such as a query pipeline, an events database, an events analyser, etc. and although the associated method include steps as explained and illustrated herein, not all of these components, configurations and steps are essential to the invention and thus should not be taken in their restrictive sense, i.e. should not be taken as to limit the scope of the present invention. It is to be understood, as also apparent to a person skilled in the art, that other suitable components and cooperations thereinbetween, as well as other suitable configurations, organizations and/or architectures may be used for the ranking system and method as described herein and as can be easily inferred herefrom, by a person skilled in the art, without departing from the scope of the invention. Moreover, the order of the steps provided herein should not be taken as to limit the scope of the invention, as the sequence of the steps may vary in a number of ways, without affecting the scope or working of the invention, as can also be understood.

[0074] The above-described embodiments are considered in all respect only as illustrative and not restrictive, and the present application is intended to cover any adaptations or variations thereof, as apparent to a person skilled in the art. Of course, numerous other modifications could be made to the above- described embodiments without departing from the scope of the invention, as apparent to a person skilled in the art.