[0001] This disclosure relates to an application spam detector that detects and penalizes potential spam applications in an application search. 5 BACKGROUND

[0002] As users are migrating to an app-centric world the demand for applications has increased, as has the supply of applications. In an app-centric world, users tend to be more inclined to use an application to perform a particular task (e.g., search for a movie time, make a dinner reservation, or play a game) instead of visiting a website that hosts a 10 web application. As the transition to an app-centric world continues, the ability to search for applications or functionality has become more important. Application search engines can search for applications to find applications for users and/or for results within applications to enhance the user experience by leveraging the functionality within applications indicated by the search results.

15

SUMMARY

[0003] One aspect of the disclosure provides a method for performing an application search. The method includes receiving, by a processing device, a search query from a remote device and determining, by the processing device, a consideration set of

20 applications based on the search query. The consideration set indicates one or more

applications corresponding to the search query. For each application indicated in the consideration set, the method includes determining, by the processing device, whether the application is a potential spam application based on one or more developer features indicating features associated with a developer of the application. When the application 25 is determined to be a potential spam application, the method includes applying a penalty to the application. The method further includes generating, by the processing device, search results based on the consideration set and any penalties applied to the one or more application indicated in the consideration set and providing, by the processing device, the search results. The search results indicate one or more of the applications indicated in the consideration set. In some implementations, the developer features include a number of source objects published by the developer and, for each of the source objects, a number of feedback units attributed to the source object.

[0004] Implementations of the disclosure may include one or more of the following 5 optional features. In some implementations, determining whether the application is a potential spam application includes determining whether the number of source objects exceeds a source object threshold. When the number of source objects exceeds the source object threshold, the method further includes determining whether a significant percentage of the source objects have less than a first threshold of feedback units

10 attributed thereto, and whether all of the source objects have less than a second threshold of feedback units attributed thereto. The second threshold is greater than the first threshold. The application is identified as a potential spam application when a significant percentage of the source objects have less than the first threshold of feedback units attributed thereto and all of the source objects have less than the second threshold of 15 feedback units attributed thereto.

[0005] In some examples, applying the penalty includes removing the application from the consideration set, such that the consideration set no longer indicates the application. The method may further include determining a score for each application indicated in the consideration set based on the search query and application data of the 20 application. Additionally or alternatively, applying the penalty may include multiplying the result score of the application by a reduction factor. Generating the search results may further include ranking the applications in the consideration set based on the respective result scores of the applications in the consideration set.

[0006] In some implementations, applying the penalty includes comparing a title of 25 the potential spam applications with the search query. The method can include applying the penalty to the potential spam application, when the title of the potential spam application does not match the search query. The method can include determining whether another application in the consideration set has a title that matches the title of the potential spam application when the title of the potential spam application does match the 30 search query. The method can include only applying the penalty when the result score of the other application is greater than the result score of the potential spam application. In some examples, providing the search results includes transmitting the search results to the remote device via a network and/feeding the search results to a subsequent search process.

[0007] Another aspect of the disclosure provides a search engine for performing an 5 application search. The search engine includes a storage device and a processing device that executes computer readable instructions. When the computer readable instructions are executed by the processing device, the processing device receives a search query from a remote device and determines a consideration set of applications based on the search query. The consideration set indicates one or more applications corresponding to the 10 search query. For each application indicated in the consideration set, the search engine includes determining whether the application is a potential spam application based on one or more developer features indicating features associated with a developer of the application. When the application is determined to be a potential spam application, the search engine includes applying a penalty to the application. The search engine further 15 generates search results based on the consideration set and any penalties applied to the one or more applications indicated in the consideration set and provides the search results. The search results indicate one or more of the applications indicated in the consideration set. In some examples, the developer features include a number of source objects published by the developer and, for each of the source objects, a number of 20 feedback units attributed to the source object.

[0008] In some implementations, determining whether the application is a potential spam application includes determining whether the number of source objects exceeds a source object threshold. When the number of source objects exceeds the source object threshold, the search engine can determine whether a significant percentage of the source 25 objects have less than a first threshold of feedback units attributed thereto. The search engine further determines whether all of the source objects have less than a second threshold of feedback units attributed thereto. The second threshold is greater than the first threshold. The application is identified as a potential spam application when a significant percentage of the source objects have less than the first threshold of feedback 30 units attributed thereto and all of the source objects have less than the second threshold of feedback units attributed thereto. [0009] In some examples, the computer-readable instructions further cause the processing device to determine a result score for each application indicated in the consideration set based on the search query and application data of the application.

Additionally or alternatively, applying the penalty may include multiplying the result 5 score of the application by a reduction factor. Generating the search results can further include ranking the applications in the consideration set based on the respective result scores of the applications in the consideration set.

[0010] In some implementations, applying the penalty includes comparing a title of the potential spam application with the search query. When the title of the potential spam 10 application does not match the search query, the search engine may apply the penalty to the potential spam application. When the title of the potential spam application does match the search query, the search engine may determine whether another application in the consideration set has a title that matches the title of the potential spam application. In some implementations, when another application in the consideration set has a title that 15 matches the title of the potential spam application, the search engine only applies the penalty when the result score of the other application is greater than the result score of the potential spam application. Providing the search results may include transmitting the search results to the remote device via a network and/or feeding the search results to a subsequent search process. 20 DESCRIPTION OF DRAWINGS

[0011] FIG. 1 is a schematic illustrating an example environment of an application search engine.

[0012] FIGS. 2A and 2B are schematics illustrating example components of an application search engine.

25 [0013] FIG. 3 is a flow chart illustrating an example set of operations for a method for performing an application search.

[0014] FIG. 4 is a flow chart illustrating an example set of operations for a method for analyzing a consideration set to identify potential spam applications.

[0015] FIG. 5 is a flow chart illustrating an example set of operations for a method 30 for analyzing a consideration set to identify potential spam applications. [0016] Like reference symbols in the various drawings indicate like elements. DETAILED DESCRIPTION

[0017] FIG. 1 illustrates an example environment 10 of an application search engine 200 in communication with a plurality of user devices 100 via a network 150 (e.g., the 5 Internet and/or one or more intranets). The application search engine 200 can include one or more computing devices (e.g., servers) that work individually or in combination to receive a query wrapper 120 containing a search query 110 from a user device 100 and to provide search results 130 corresponding to the search query 110 to the user device 100.

[0018] An application 264 can refer to computer readable instructions that cause a 10 user device 100 to perform a task. In some examples, an application 264 is referred to as an "app" or a "program." Example applications 264 include, but are not limited to, word processing applications, spreadsheet applications, messaging applications, media streaming applications, social networking applications, and games.

[0019] Applications 264 can be executed on a variety of different user devices 100. 15 For example, applications 264 can be executed on mobile computing devices, such as smart phones 100b, tablets 100a, and wearable computing devices (e.g., headsets and/or watches). Applications 264 can also be executed on other types of user devices 100 having other form factors, such as laptop computers 100c, desktop computers, or other consumer electronic devices. In some examples, applications 264 are installed on a user 20 device 100 prior to the purchase of the user device 100. In other examples, a user may download and install applications 264 on the user device 100.

[0020] In some implementations, the functionality of an application 264 is accessed on the user device 100 on which the application 264 is installed. In some examples, the user device on which the application 264 is installed can execute all of an application's 25 functionality. These applications 264 may function without communication with other computing devices. In other examples, an application 264 installed on a computing device may access information from other remote computing devices during operation. For example, a weather application installed on a computing device may access the latest weather information via the remote server and display the accessed weather information 30 to the user through the installed weather application. In still other examples, a web application is an application 264 that may be partially executed by a web browser executing on the user's computing device and partially by a remote server. For example, a web application may be an application that is executed, at least in part, by a web server and accessed by a web browser of the user's computing device. Example web

5 applications may include, but are not limited to, web-based email, online auctions, and online retail sites.

[0021] A user device 100 receives a search query 110 from a user via a user interface of the device 100. A search query 110 can include one or more search terms. The user, for example, can provide the search terms by typing text containing the search terms via a 10 touch screen keyboard or can provide speech input containing the search terms via a

microphone of the user device 100. In the latter scenario, the user device 100 can perform speech-to-text conversion to identify the search terms. The user device 100 can generate a query wrapper 120 that contains the search query 110. A query wrapper 120 is a data unit that communicates with the application 264 search server 200 via a network 15 150. The query wrapper 120 can further include one or more query parameters 115. For example, a query wrapper 120 can include query parameters 115 that indicate one or more of a geolocation of the user device 100, a username associated with the user device 100, and an operating system of the user device 100. In some implementations a search application 264 executing on the user device 100 receives the search query 110 (e.g., via 20 a graphical user interface of the search application or via a search bar), determines zero or more query parameters 115, generates the query wrapper 120 based on the search query 110 and the query parameters 115, and transmits the query wrapper 120 to the application search engine 200. In some implementations, the user device 100 transmits the query wrapper 120 to a partner device (e.g., a server of an organization affiliated in some way 25 with the application search engine 200), and the partner device forwards the query

wrapper 120 to the application search engine 200.

[0022] The application search engine 200 receives and processes the query wrapper 120. Additionally, the application search engine 200 generates the search results 130 based on the contents of the query wrapper 120. The search results 130 can identify one 30 or more applications 264 that are relevant to the search query 110. The application search engine 200 can perform different types of application searches. Examples of application searches are function based searches and deep searches.

[0023] In some implementations, the application search engine 200 performs function based searches. In these searches, the application search engine 200 returns search 5 results 130 that indicate applications 264 that can perform one or more functions

implicated by the search query 110 and/or the query parameters 115. For example, in response to a search query 110 containing“organize my finances,” the application search engine 200 can provide search results 130 indicating one or more applications 264 that have functionality directed to organizing the finances of a user. In another example, the 10 search query 110 may contain the search terms“games for my five year old.” In

response to this example search query 110, the application search engine 200 can return search results 130 indicating one or more applications 264 that are age appropriate for younger children.

[0024] In some implementations, the application search engine 200 performs deep 15 searching. In these implementations, the application search engine 200 can provide

search results 130 that provide one or more links to specific states of one or more applications, whereby the specific states are pertinent to the search query 110 and/or the query parameters 115. The links can be displayed on the user device 100 such that when the link is selected by a user, the user device 100 launches the application 264 to the 20 specific state indicated by the link. For example, in response to a query wrapper 120 containing the search terms“Thai food” and a query parameter 115 indicating the current location of the user device 100, the applications search engine 200 can provide search results 130 including a first link to an entry for a local Thai restaurant in a restaurant review application (e.g., the YELP® application by Yelp, Inc.) and a second link to an 25 entry to make restaurant reservations (e.g., the OPENTABLE® application by OpenTable Inc.) at a local Thai restaurant. In these types of searches, the application search engine 200 can perform a function based search at the front end of the deep search to identify applications 264 that are relevant to the search query 110 and/or the query parameters 115. The search results 130 of the function based search can be further processed to 30 identify the results of the deep search. [0025] The foregoing examples of application searches are provided for explanation only and to provide context. The application search engine 200 may be configured to perform other types of searches as well. For instance, in some applications the search engine 200 may be configured to perform a keyword search, whereby the application 5 search engine 200 searches for applications 264 having titles that match the search terms contained in the search query 110. Additionally or alternatively, the application search engine 200 may perform an entity search, whereby the application search engine 200 identifies applications 264 that are relevant to entity types and/or entity names that are indicated in the search query 110.

10 [0026] A recent development has been for application developers (malicious and otherwise) to develop“spam” applications that provide little or minimal functionality in order to appear in application search results 130 and to entice users to download the spam applications to their respective user devices 100. The motivations for developing and selling spam applications include gaining access to sensitive user information (e.g., the 15 location of the user device 100, the purchasing habits of a user, or financial information of the user) and/or selling the spam applications to the user and to collect the fees associated with the sale.

[0027] In order to reduce the amount of spam applications 264 that are indicated in the search results 130, the application search engine 200 can be configured with an 20 application spam detector 228 to identify potential spam applications 264s. In some implementations, the application search engine 200 identifies a consideration set 140 of applications 264, 264c that indicates one or more applications 264, 264c that are relevant to the search query 110 and/or the query parameters 115. The application search engine 200 identifies the applications 264c that are indicated in the search results 130 from the 25 consideration set 140. The application spam detector 228 analyzes developer features of each application 264c indicated in the consideration set 140 to determine whether the application 264c is a potential spam application 264s. When the application spam detector 228 identifies a potential spam application 264s, the application spam detector 228 can apply a penalty to the potential spam application 264s. A penalty can include 30 removing the potential spam application 264s from the search results 130 or by adjusting a result score (discussed in further detail below) of the potential spam application 264s. In some implementations, when the application search engine 200 identifies a potential spam application 264s, the application search engine 200 can identify the potential spam application 264s as such in an application datastore 260. Further, in some

implementations, the application search engine 200 can identify applications 264 related 5 to the potential spam applications 264s also as being potential spam applications 264s (e.g., applications published by the same developer).

[0028] FIGS. 2A and 2B illustrate an example application search engine 200 configured to reduce the amount of spam applications 264 appearing in search results 130. As previously mentioned, the application search engine 200 may be implemented as 10 a single computing device or a plurality of computing devices that operate in a distributed or individual manner. The application search engine 200 can include, but is not limited to a processing device 210, a network interface device 240, and a storage device 250. The application search engine 200 can include one or more other components not explicitly shown in FIG. 2A.

15 [0029] The processing device 210 can include memory (e.g., RAM and/or ROM) that stores computer readable instructions and one or more physical processors that execute the computer readable instructions. In implementations where the processing device 210 includes more than one processor, the processors can operate in an individual or distributed manner. Furthermore, in these implementations the processors can be in the 20 same computing device or can execute in separate computing devices (e.g., rack-mounted servers). The processing device 210 can execute a search module 220.

[0030] The network interface device 240 includes one or more devices that can perform wired or wireless (e.g., WiFi or cellular) communication via the network 150. Examples of the network interface device 240 include, but are not limited to, a

25 transceiver configured to perform communications using the IEEE 802.11 wireless

standard, an Ethernet port, a wireless transmitter, and a universal serial bus (USB) port.

[0031] The storage device 250 can include one or more computer readable storage mediums (e.g., hard disk drives and/or flash memory drives). The storage mediums can be located at the same physical location or device or at different physical locations or 30 devices. The storage device 250 can store an application datastore 260. [0032] The application datastore 260 may include one or more databases, indices (e.g., inverted indices), files, or other data structures storing this data. The application datastore 260 includes application data of different applications. The application data of an application may include keywords associated with the application, reviews associated 5 with the application, the name of the developer of the application, the platform of the application, the price of the application, application statistics (e.g., a number of downloads of the application and/or a number of ratings of the application), a category of the application, and other information. The application data of an application may also include an indicator (e.g., a flag) that indicates whether the application has been

10 determined to be a potential spam application. The application datastore 260 may include metadata for a variety of different applications available on a variety of different operating systems. The search module 220 may retrieve and analyze the application data from the application datastore 260 in order to perform application searches and to provide search results 130.

15 [0033] In some implementations, the application datastore 260 stores the application data in application records 262. Each application record 262 can correspond to an application 264 and may include the application data pertaining to the application 264. An example application record 262 includes an application name, an application identifier, and other application features. The application record 262 may generally 20 represent the application data stored in the application datastore 260 that is related to an application 264.

[0034] The application name may be the name of the application 264 represented by the data in the application record 262. Example application names may include

"FACEBOOK®" by Facebook, Inc., "TWITTER®" by Twitter, Inc., or "MICROSOFT 25 WORD®" by Microsoft Corp. The application identifier (hereinafter "application ID") identifies the application record 262 amongst the other application records 262 included in the application datastore 260. In some implementations, the application ID uniquely identifies the application record 262. The application ID may be a string of alphabetic, numeric, and/or symbolic characters (e.g., punctuation marks) that uniquely identify the 30 application 264 represented by the application record 262. In some implementations, the application ID is the ID assigned to the application 264 by a digital distribution platform (e.g., the APP STORE® digital distribution platform by Apple Inc. or the GOOGLE PLAY® digital distribution platform by Google, Inc.) that offers the application 264. In other implementations, the application search engine 200 assigns application IDs to each application 264 when creating an application record 262 for the application 264.

5 [0035] The application features may include any type of data that may be associated with the application 264 represented by the application record 262. The application features may include a variety of different types of metadata. For example, the application features may include structured, semi-structured, and/or unstructured data. The application features may include information that is extracted or inferred from 10 documents retrieved from other data sources (e.g., application digital distribution

platforms, application developers, blogs, and reviews of applications) or that is manually generated (e.g., entered by a human). The application features may be updated so that up to date results can be provided in response to a search query 110.

[0036] The application features may include the name of the developer of the

15 application 264, a category (e.g., genre) of the application 264, a description of the

application 264 (e.g., a description provided by the developer), a version of the application 264, the operating system the application 264 is configured for, and the price of the application 264. The application features further include feedback units provided to the application 264. Feedback units can include ratings provided by reviewers of the 20 application 264 (e.g., four out of five stars) and/or textual reviews (e.g.,“This app is

great”). The application features can also include application statistics. Application statistics may refer to numerical data related to the application 264. For example, application statistics may include, but are not limited to, a number of downloads, a download rate (e.g., downloads per month), and/or a number of feedback units (e.g., a 25 number of ratings and/or a number of reviews). The application features may also

include information retrieved from websites, such as comments associated with the application 264, articles associated with the application 264 (e.g., wiki articles), or other information. The application features may also include digital media related to the application 264, such as images (e.g., icons associated with the application 264 and/or 30 screenshots of the application 264) or videos (e.g., a sample video of the application 264). [0037] In some implementations, an application record 262 can group one or more application editions under a canonical application. The term canonical application can refer to a representative of a group of similar application editions. Put another way, the canonical application may be a representative or umbrella under which the one or more 5 application editions are organized. Each application edition in a group of application editions that are organized under the same canonical application respectively indicate an application that is functionally similar to other application editions organized under the canonical application. In other words, the application editions can represent different versions of the same application (e.g., the canonical application). Examples of edition 10 applications are different language versions of an application 264 (e.g., English, Russian, or Chinese), different platform versions of an application 264 (e.g., IOS® by Apple, Inc., or ANDROID® by Google Inc.), different functional characteristics (e.g., a light version, an advanced user version), and different resolution versions of an application 264 (e.g., standard definition or high definition). It is noted that applications 264 which are related 15 but not similar, e.g., sequels in a series of games or a different release of a serial

application, are typically not classified under the same canonical application. For example, ANGRY BIRDS®, by Rovio Entertainment Ltd. may be a separate canonical application from ANGRY BIRDS® RIO. Although the applications 264 may have a same underlying concept, they are separate games.

20 [0038] The application search module 220 receives query wrappers 120 from remote user devices 100 and generates search results 130 based on the query wrapper 120 and the application data stored in the application datastore 260. FIG. 2B illustrates an example application search module 220. In the illustrated example, the application search module 220 includes a query analysis module 222, a set generation module 224, a 25 set processing module 226, and an application spam detector 228.

[0039] The query analysis module 222 receives the search wrapper 120 that contains a search query 110. The query analysis module 222 may perform various analysis operations on the search query 110. For example, analysis operations performed by the query analysis module 222 may include, but are not limited to, tokenization of the search 30 query 110, filtering of the search query 110, stemming the search query 110,

synonomyzation of the search query 110, and stop word removal. In some implementations, the query analysis module 222 further generates one or more reformulated search queries based on the search query 110 and the query parameters 115. Reformulated search queries are search queries that are based on some subcombination of the search query 110 and the query parameters 115.

5 [0040] The set generation module 224 identifies a consideration set 140 of

applications 264, 264c (e.g., a list of applications) based on the search query 110 and, in some implementations, the reformulated subqueries. In some examples, the set generation module 224 identifies the consideration set 140 by identifying applications 264 that correspond to the search query 110 or the reformulated search queries based on 10 matches between terms of the query 110 and terms in the application data of the

application 264 (e.g., in the application record 262 of the application 264). For example, the set generation module 224 may identify one or more applications 264, represented in the application datastore 260 based on matches between tokens generated by the query analysis module 222 and words included in the application records 262 of those

15 applications 264. The consideration set 140 may include a list of application IDs and/or a list of application names.

[0041] The set processing module 226 performs a variety of different processing operations on the consideration set 140. In some implementations, the set processing module 226 generates a result score for each of the applications 264c indicated in the 20 consideration set 140. In some examples, the set processing module 226 culls the

consideration set 140 based on the result scores of the applications 264c contained therein. For example, the set processing module 226 may remove applications 264c from the consideration set 140 that have result scores that do not exceed a threshold.

Additionally or alternatively, the set processing module 226 may rank the applications 25 264c indicated in the consideration set 140 based on the respective result scores of each application 264c. The information conveyed in the search results 130 may depend on how the result scores are calculated by the set processing module 226. For example, the result scores may indicate the relevance of an application 264, 264c to the search query 110, the popularity of an application 264, 264c in the marketplace, the quality of an 30 application 264, 264c, or other properties of the application 264, 264c, depending on what parameters the set processing module 226 uses to score the applications 264, 264c. [0042] The set processing module 226 may generate result scores for applications 264 in a variety of different ways. In general, the set processing module 226 may generate a result score for an application 264 based on one or more scoring features. The scoring features may be associated with the application 264 and/or the query 110. An 5 application scoring feature may include any data associated with an application 264. For example, application scoring features may include any of the application features included in the application record 262 or any additional parameters related to the application 264, such as data indicating the popularity of an application 264 (e.g., number of downloads) and the ratings (e.g., number of stars) associated with an application 264. 10 A query scoring feature may include any data associated with a query 110. For example, query scoring features may include, but are not limited to, a number of words in the query 110, the popularity of the query 110, and the expected frequency of the words in the query 110. An application-query scoring feature may include any data, which may be generated based on data associated with both the application 264 and the query 110 (e.g., 15 the query 110 that resulted in identification of the application record 262 by the set

generation module 224). For example, application-query scoring features may include, but are not limited to, parameters that indicate how well the terms of the query 110 match the terms of the identified application record 262. The set processing module 226 may generate a result score for an application 264 based on at least one of the application 20 scoring features, the query scoring features, and the application-query scoring features.

[0043] The set processing module 226 may determine a result score based on one or more of the scoring features listed herein and/or additional scoring features not explicitly listed. In some examples, the set processing module 226 includes one or more machine learned models (e.g., a supervised learning model) configured to receive one or more 25 scoring features. The one or more machine learned models may generate result scores based on at least one of the application scoring features, the query scoring features, and the application-query scoring features. For example, the set processing module 226 may pair the query 110 with each application 264 and calculate a vector of features for each (query 110, application 264) pair. The vector of features may include application scoring 30 features, query scoring features, and application-query scoring features. The set

processing module 226 may then input the vector of features into a machine-learned regression model to calculate a result score that may be used to rank the applications 264c in the consideration set 140. The foregoing is an example of how a result score is calculated. Some implementations may calculate the result score in a different manner.

[0044] The application search engine 200 may use the result scores in a variety of 5 different ways. In some examples, the application search engine 200 uses the result scores to rank the applications 264 in the search results 130 that are presented on a user device 100. In these examples, a greater result score may indicate that the application 264c is more relevant to the search query 110 and/or the query parameters 115 than an application 264c having a lesser result score. In examples where the search results 130 10 are displayed as a list of application descriptions (e.g., an icon of an application 264 and a description of the application 264) on a user device 100, the application descriptions associated with larger result scores may be listed nearer to the top of the results list (e.g., near to the top of the screen). In these examples, application descriptions having lesser result scores may be located farther down the list (e.g., off screen) and may be accessed 15 by a user scrolling down the screen of the user device 100 or viewing a subsequent page of search results 130.

[0045] The application spam detector 228 analyzes the application data of the applications 264c indicated in the consideration set 140 and identifies potential spam applications 264c, 264s based on the analysis. The application spam detector 228 can 20 consider the developer of an application 264c and features of the developer when

analyzing the application 264c. In some examples, features of a developer can include, but are not limited to, the number of source objects published by the developer and the number of feedback units for each of the source objects. Once the application spam detector 228 identifies a potential spam application 264s, the application spam detector 25 228 can apply a penalty to the potential spam application 264s.

[0046] A source object can refer to instances of applications 264 made available by the developer. For instance, if the developer sells the same application edition on two different digital distribution platforms, each instance of the application edition can be considered a source object. In other implementations, each application edition published 30 by the developer is counted as a single source object regardless of how many application digital distribution platforms offer the application edition. The number of source objects also includes application editions organized under different canonical applications (i.e., applications 264 that are not considered similar) but published by the developer. For example, if three different versions of a first application are offered on the GOOGLE PLAY® digital distribution platform and two different versions of a second application 5 published by the developer of the first application are offered on the GOOGLE PLAY® digital distribution platform, then the application spam detector 228 counts five source objects. Furthermore, in implementations where applications 264 are not organized as editions under canonical applications, the number of source objects can refer to the total number of application instances published by the developer, regardless of whether the 10 applications 264 are similar or not. Regardless of how the application spam detector 228 counts source objects, the manner by which the application spam detector 228 counts source objects must be consistent.

[0047] In some implementations, the application spam detector 228 analyzes the application data of each application 264c indicated in the consideration set 140 to 15 determine whether the application 264c is a potential spam application 264s. For each application 264c, the application spam detector 228 can identify a developer of the application 264c and then identify all source objects published by the developer. If the number of source objects attributed to the developer exceeds a source object threshold (e.g., more than 30 source objects), the application spam detector 228 can analyze the 20 amount of feedback units provided for each application 264c to determine whether the application 264c is a potential spam application 264s. In some implementations, the application spam detector 228 determines whether: i) a significant percentage of the source objects (e.g., more than 90% of the source objects) received less than a first threshold of feedback units (e.g., less than 10 feedback units); and ii) any of the source 25 objects received more than a second threshold of feedback units (e.g., more than 300 feedback units). If a significant percentage of source objects received less feedback units than first threshold of feedback units and no source object received more feedback units than the second threshold of feedback units, then the application spam detector 228 identifies the application 264s as a potential spam application 264s. The values provided 30 above are for example only. A significant percentage can refer to a percentage that exceeds a threshold. In the example discussed above, the threshold defining the significant percentage is 90%. The application spam detector 228 can utilize other suitable thresholds for defining the significant percentage.

[0048] When the application spam detector 228 identifies an application 264c as a potential spam application 264s, the application spam detector 228 applies a penalty to 5 the potential spam application 264s. A penalty can be any action that impacts how the application 264s is represented in the search results 130. In some implementations, the penalty is removing the potential spam application 264s from the search results 130. In other implementations, the application spam detector 228 penalizes the potential spam application 264s by multiplying the result score of the application 264s by a reduction 10 factor (e.g., .05). This type of penalty may result in the application 264s being removed from the search results 130 or being listed at the end of the search results 130 due to its lowered result score. The reduction factor can be hard coded by a developer or can be learned by a machine learner. In some implementations, the application spam detector 228 can further penalize the potential spam application by identifying the potential spam 15 application 264s and each of the source objects corresponding to the potential spam

application 264s as potential spam applications 264s in their respective application data. In these implementations, the application spam detector 228 can update the application records 262 of the applications 264 corresponding to each of the source objects (they are listed as applications 264 in the application datastore 260) to indicate that the applications 20 264 are potential spam applications 264s.

[0049] In some implementations, the application spam detector 228 does not apply the penalty when the title of the potential spam application 264s matches the search terms of the search query 110. In some of these implementations, the application spam detector 228 applies the penalty under this condition if there are one or more higher scoring 25 applications 264 that share a title with the potential spam application 264s. In these

implementations, the application spam detector 228 compares the title of the potential spam application 264s with the search terms. In some implementations, the application spam detector 228 compares the title of the potential spam application 264s and the search terms by determining whether the search terms and the title have matching name 30 norms. A name norm can refer to the result of a function that converts a set of terms into a standardized representation. In some implementation, the application spam detector 228 implements an intelligent domain specific normalization function to determine the name norms of the search terms and the terms of the application title. The function can downcase the terms, remove punctuation, and/or remove any unnecessary words from the terms to obtain the name norms. If the name norms match, the application spam detector 5 228 reviews the consideration set 140 to determine whether there is another application 264c indicated in the consideration set 140 that shares a title with the potential spam application 264c and has a greater result score than the potential spam application 264s. If the consideration set 140 lists an application 264c that shares a title with the potential spam application 264s and that has a higher result score than the potential spam

10 application 264s, the application spam detector 228 applies the penalty to the potential spam application 264s. If the consideration set 140 does not list an application 264c that shares a title with the potential spam application 264s or if the potential spam application 264s has a higher result score than any other application 264c with which it shares a title, the application spam detector 228 does not apply the penalty to the potential spam 15 application 264s.

[0050] Once the application spam detector 228 has analyzed the applications 264c listed in the consideration set 140, the results generation module 230 can generate the search results 130 and provide the search results 130 to the user device 100. In some implementations, the results generation module 230 provides the search results 130 to the 20 user device 100 for display to a user. In these implementations, generating the search results 130 can include ranking the applications 264c in the consideration set 140 based on their respective result scores and obtaining display items (e.g., application icon, application ratings, application reviews, and/or application descriptions) corresponding to each application 264c in the consideration set 140. When ranking the applications 264c 25 in the consideration set 140, any penalties applied by the application spam detector 228 are taken into consideration by way of the result score (or alternatively, by the exclusion of the potential spam application 264s from the consideration set 140). The results generation module 230 can include the display items in the search results 130 such that the user device 100 can display the search results 130 via its user interface. The results 30 generation module 230 transmits the search results 130 to the user device 100 via the network 150. The user device100 can render and display the search results 130. In other implementations, the results generation module 230 can provide the search results 130 to another component of the search engine 200 that performs additional searches. These additional searches may include a deep search. In these implementations, the results generation module 230 can rank the applications 264c indicated in the consideration set 5 140 based on their respective result scores and can communicate the ranked consideration set 140 (or a subset of the consideration set 140 containing the highest ranked scores) to the subsequent search process.

[0051] The description of the application search engine 200 is provided for example only. The application spam detector 228 described above can be used in any other type 10 of application search.

[0052] FIG. 3 illustrates an example set of operations for a method 300 for performing a search based on a received query wrapper 120. The method 300 is described with reference to the application search module 210 of FIG. 2. The method 300 may, however, be executed by any other suitable component.

15 [0053] In block 302, the query analysis module 222 receives a query wrapper 120 containing a search query 110. The query wrapper 120 may further include one or more query parameters 115 (e.g., location of the user device 100, platform of the user device 100, or a username). In block 304, the query analysis module 222 analyzes the query wrapper 120. The query analysis module 222 can reformulate the search query 110 to 20 include one or more of the query parameters 115. Furthermore, in some

implementations, the query analysis module 222 can generate a plurality of reformulated search queries based on the search query 110 and the query parameters 115.

Reformulated search queries are sub-combinations of the search terms contained in the search query 110 and/or the query parameters 115. For instance, if the query wrapper 25 120 contained in the search query 110 is“find late night dining options,” and the query parameters 115 included a location corresponding to Detroit, MI, the reformulated search queries could include“late night dining,”“late night dining Detroit,”“dining Detroit,” “dining,”“late night,”“late night Detroit MI,” etc.

[0054] In block 306, the set generation module 224 identifies a consideration set 140 30 of applications 264c based on the search query 110 (e.g., based on output from the query analysis module 222) and/or the reformulated search queries. In some implementations, the set generation module 224 identifies any application record 262 that contains the terms in the search query 110 and/or the reformulated search queries. If an application record 262 contains one or more of these terms, the application 264c indicated by the application record 262 is included in the consideration set 140.

5 [0055] In block 308, the set processing module 226 processes the consideration set 140 of applications 264c. In some implementations, the set processing module 226 determines a result score for each of the applications 264c in the consideration set 140. As previously discussed, the set processing module 226 can implement one or more machine learned models to score each application 264c in the consideration set 140. 10 [0056] In block 310, the application spam detector 228 analyzes the consideration set 140 to determine whether any of the applications 264c included therein are potential spam applications 264s. FIG. 4 illustrates an example set of operations for a method 400 for determining whether an application 264, 264c is a potential spam application 264s. The method 400 can be executed for each application 264c in the consideration set 140. 15 [0057] At block 402, the application spam detector 228 obtains the application data of the application 264c that it is analyzing. In some implementations, the application spam detector 228 obtains the application record 262 of the application 264c. At block 404, the application spam detector 228 determines whether the application 264c had been previously identified as a potential spam application 264s. If so, the application spam 20 detector 228 applies a penalty to the application spam detector 228, as shown at block 414. As previously discussed, the application spam detector 228 can either multiply the result score of the application 264s by a reduction factor (e.g., .05 or .01) or can remove the application 264s from the search results 130 altogether.

[0058] If the application 264 has not been previously identified as a potential spam 25 application 264s, the application spam detector 228 obtains the features of the developer of the application 264, as shown at block 406. In some implementations, the application spam detector 228 identifies the application developer of the application 264 from the application record 262. The application spam detector 228 searches the application datastore 260 for all other applications 264 published by the developer and retrieves their 30 respective application records 262. The application spam detector 228 can determine the number of source objects corresponding to the developer based on the information contained in the application records 262. The retrieved application records 262 also include the statistics of each application 264, including the number of feedback units that each application 264 has received. In this way, the application spam detector 228 can identify the number of feedback units attributed to each of the source objects.

5 [0059] The application spam detector 228 determines whether the application 264, 264c under consideration is a potential spam application 264s based on the features of the developer. At block 408, the application spam detector 228 determines whether the number of source objects attributed to the developer exceeds a source object threshold (e.g., more than 30 source objects). If the number of source objects is less than the 10 source object threshold, the application 264c is not identified as a potential spam

application 264s, and the application spam detector 228 can stop analyzing the application 264c.

[0060] If the number of source objects does exceed the source object threshold, the application spam detector 228 determines whether a significant percentage of the source 15 objects (e.g., more than 90% of the source objects) received less than a first threshold of feedback units (e.g., less than 10 ratings), as shown at block 410. If less than a significant percentage of source objects received fewer feedback units than the first threshold, the application 264s is not identified as a potential spam application 264s, and the application spam detector 228 can stop analyzing the application 264c.

20 [0061] If a significant percentage of source objects received less feedback units than the first threshold, the application spam detector 228 can determine whether any one of the source objects received more feedback units than a second threshold (e.g., more than 300 ratings), as shown at block 412. If none of the source objects received more feedback units than the second threshold, the application 264c is not identified as a 25 potential spam application 264s and the application spam detector 228 can stop analyzing the application 264. Otherwise, the application 264c is identified as a potential spam application 264s. When the application spam detector 228 identifies the application 264c as a potential spam application 264s, the application spam detector 228 can apply a penalty to the application 264s, as shown at block 414.

30 [0062] The method 400 of FIG. 4 is provided for example only. The ordering of the blocks is not mandatory. Furthermore, the values of the thresholds are provided for example and not intended to limit the scope of the disclosure. Variations of the method 400 are contemplated and within the scope of the disclosure.

[0063] Referring back to FIG. 3, at block 312, the application search module 210 generates and provides the search results 130. In some implementations, the results 5 generation module 230 generates search results 130 that it transmits to the user device 100 that provided the search wrapper 120. In these implementations, the results generation module 230 can rank the applications 264c remaining in the consideration set 140 based on the result scores of the applications 264c. The results generation module 230 also obtains display items (e.g., application name, icons, descriptions, screenshots) 10 for each of the applications 264 and formats the search results 130. The results

generation module 230 can then transmit the search results 130 to the user device 100. In other implementations, the results generation module 230 provides the search results 130 to another search process. In these implementations, the results generation module 230 ranks the applications 264c listed in the consideration set 140 and provides the search 15 results 130 to the other search process (e.g., a deep search process).

[0064] FIG. 5 illustrates an example set of operations for a method 500 for analyzing a consideration set 140 to identify potential spam applications 264s of performing a search based on a received query wrapper 120. The method 500 is described with reference to the application search module 210 of FIG. 2. The method 500 may, 20 however, be executed by any other suitable component.

[0065] In block 502, the method 500 includes receiving, by the processing device 210, a search query 110 from a remote device 100, 100a-100c and determining a consideration set 140 of applications 264, 264c based on the search query 110. The consideration set 140 indicates one or more applications 264, 264c corresponding to the 25 search query 110. For each application 264, 264c indicated in the consideration set 140, in block 506, the method 500 includes determining whether the application 264, 264c is a potential spam application 264s based on one or more developer features indicating features associated with a developer of the application 264, 264c. When the application 264, 264c is determined to be a potential spam application 264s, in block 508, the method 30 500 includes applying a penalty to the application 264, 264c. In block 510, the method 500 further includes generating, by the processing device 210, search results 130 based on the consideration set 140 and any penalties applied to the one or more application 264, 264c indicated in the consideration set 140 and at block 512, providing the search results 130. The search results 130 indicate one or more of the applications 264 indicated in the consideration set 140. In some implementations, the developer features include a number 5 of source objects published by the developer and, for each of the source objects, a

number of feedback units attributed to the source object.

[0066] In some implementations, determining whether the application 264 is a potential spam application 264s includes determining whether the number of source objects exceeds a source object threshold. When the number of source objects exceeds 10 the source object threshold, the method 500 further includes determining whether a

significant percentage of the source objects have less than a first threshold of feedback units attributed thereto, and whether all of the source objects have less than a second threshold of feedback units attributed thereto. The second threshold is greater than the first threshold. The application 264 is identified as a potential spam application 264s 15 when a significant percentage of the source objects have less than the first threshold of feedback units attributed thereto and all of the source objects have less than the second threshold of feedback units attributed thereto.

[0067] In some examples, applying the penalty includes removing the application 264 from the consideration set 140, such that the consideration set 140 no longer indicates the 20 application 264. The method 500 may further include determining a score for each

application 264 indicated in the consideration set 140 based on the search query 110 and application data of the application 264. Additionally or alternatively, applying the penalty may include multiplying the result score of the application 264 by a reduction factor. Generating the search results 130 may further include ranking the applications 25 264 in the consideration set 140 based on the respective result scores of the applications 264 in the consideration set 140.

[0068] In some implementations, applying the penalty includes comparing a title of the potential spam applications 264s with the search query 110. The method 500 can include applying the penalty to the potential spam application 264s, when the title of the 30 potential spam application 264s does not match the search query 110. The method 500 can include determining whether another application 264 in the consideration set 140 has a title that matches the title of the potential spam application 264s when the title of the potential spam application 264s does match the search query 110. The method 500 can include only applying the penalty when the result score of the other application 264 is greater than the result score of the potential spam application 264s. In some examples, 5 providing the search results 130 includes transmitting the search results 130 to the remote device 100, 100a-100c via a network and/feeding the search results 130 to a subsequent search process.

[0069] The implementations of the systems and techniques described here can be realized in digital electronic and/or optical circuitry, integrated circuitry, specially 10 designed ASICs (application specific integrated circuits), computer hardware, firmware, software, and/or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and/or

interpretable on a programmable system including at least one programmable processor, which may be special or general purpose, coupled to receive data and instructions from, 15 and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0070] These computer programs (also known as programs, software, software applications or code) include machine instructions for a programmable processor, and can be implemented in a high-level procedural and/or object-oriented programming language, 20 and/or in assembly/machine language. As used herein, the terms“machine-readable medium” and“computer-readable medium” refer to any computer program product, non- transitory computer readable medium, apparatus and/or device (e.g., magnetic discs, optical disks, memory, Programmable Logic Devices (PLDs)) used to provide machine instructions and/or data to a programmable processor, including a machine-readable 25 medium that receives machine instructions as a machine-readable signal. The term

“machine-readable signal” refers to any signal used to provide machine instructions and/or data to a programmable processor.

[0071] Implementations of the subject matter and the functional operations described in this specification can be implemented in digital electronic circuitry, or in computer 30 software, firmware, or hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them. Moreover, subject matter described in this specification can be implemented as one or more computer program products, i.e., one or more modules of computer program instructions encoded on a computer readable medium for execution by, or to control the operation of, data processing apparatus. The computer readable medium can be a machine-readable 5 storage device, a machine-readable storage substrate, a memory device, a composition of matter effecting a machine-readable propagated signal, or a combination of one or more of them. The terms“data processing apparatus”,“computing device” and“computing processor” encompass all apparatus, devices, and machines for processing data, including by way of example a programmable processor, a computer, or multiple processors or 10 computers. The apparatus can include, in addition to hardware, code that creates an

execution environment for the computer program in question, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them. A propagated signal is an artificially generated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal 15 that is generated to encode information for transmission to suitable receiver apparatus.

[0072] A computer program (also known as an application, program, software, software application, script, or code) can be written in any form of programming language, including compiled or interpreted languages, and it can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other 20 unit suitable for use in a computing environment. A computer program does not

necessarily correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files that store one or more modules, sub programs, or portions of 25 code). A computer program can be deployed to be executed on one computer or on

multiple computers that are located at one site or distributed across multiple sites and interconnected by a communication network.

[0073] The processes and logic flows described in this specification can be performed by one or more programmable processors executing one or more computer programs to 30 perform functions by operating on input data and generating output. The processes and logic flows can also be performed by, and apparatus can also be implemented as, special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit).

[0074] Processors suitable for the execution of a computer program include, by way of example, both general and special purpose microprocessors, and any one or more 5 processors of any kind of digital computer. Generally, a processor will receive

instructions and data from a read only memory or a random access memory or both. The essential elements of a computer are a processor for performing instructions and one or more memory devices for storing instructions and data. Generally, a computer will also include, or be operatively coupled to receive data from or transfer data to, or both, one or 10 more mass storage devices for storing data, e.g., magnetic, magneto optical disks, or optical disks. However, a computer need not have such devices. Moreover, a computer can be embedded in another device, e.g., a mobile telephone, a personal digital assistant (PDA), a mobile audio player, a Global Positioning System (GPS) receiver, to name just a few. Computer readable media suitable for storing computer program instructions and 15 data include all forms of non-volatile memory, media and memory devices, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks; magneto optical disks; and CD ROM and DVD-ROM disks. The processor and the memory can be supplemented by, or incorporated in, special purpose logic circuitry.

20 [0075] To provide for interaction with a user, one or more aspects of the disclosure can be implemented on a computer having a display device, e.g., a CRT (cathode ray tube), LCD (liquid crystal display) monitor, or touch screen for displaying information to the user and optionally a keyboard and a pointing device, e.g., a mouse or a trackball, by which the user can provide input to the computer. Other kinds of devices can be used to 25 provide interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, speech, or tactile input. In addition, a computer can interact with a user by sending documents to and receiving documents from a device that is used by the user; for 30 example, by sending web pages to a web browser on a user's client device in response to requests received from the web browser. [0076] One or more aspects of the disclosure can be implemented in a computing system that includes a backend component, e.g., as a data server, or that includes a middleware component, e.g., an application server, or that includes a frontend

component, e.g., a client computer having a graphical user interface or a Web browser 5 through which a user can interact with an implementation of the subject matter described in this specification, or any combination of one or more such backend, middleware, or frontend components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks 150 include a local area network (“LAN”) and a wide area 10 network (“WAN”), an inter-network (e.g., the Internet), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks).

[0077] The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs 15 running on the respective computers and having a client-server relationship to each other.

In some implementations, a server transmits data (e.g., an HTML page) to a client device (e.g., for purposes of displaying data to and receiving user input from a user interacting with the client device). Data generated at the client device (e.g., a result of the user interaction) can be received from the client device at the server.

20 [0078] While this specification contains many specifics, these should not be

construed as limitations on the scope of the disclosure or of what may be claimed, but rather as descriptions of features specific to particular implementations of the disclosure. Certain features that are described in this specification in the context of separate implementations can also be implemented in combination in a single implementation. 25 Conversely, various features that are described in the context of a single implementation can also be implemented in multiple implementations separately or in any suitable sub- combination. Moreover, although features may be described above as acting in certain combinations and even initially claimed as such, one or more features from a claimed combination can in some cases be excised from the combination, and the claimed 30 combination may be directed to a sub-combination or variation of a sub-combination. [0079] Similarly, while operations are depicted in the drawings in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. In certain circumstances, multi-tasking and 5 parallel processing may be advantageous. Moreover, the separation of various system components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

10 [0080] A number of implementations have been described. Nevertheless, it will be understood that various modifications may be made without departing from the spirit and scope of the disclosure. Accordingly, other implementations are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve desirable results.

15