[0001] Data sets can be divided into a number of categories.

Categories can describe similarities between data instances in data sets. Categories can be used to analyze data sets. The discovery of new similarities between data instances can lead to the creation of new categories.

[00023 Figure 1 illustrates a block diagram of an example of a computing device according to the present disclosure.

[0003] Figure 2A illustrates a diagram of an example of a number of data sets according to the present disclosure.

[0004] Figure 2B illustrates a diagram of an example of a number of data sets according to the present disclosure.

[0005] Figure 3 illustrates a block diagram of an example of a system for residual data identification according to the present disclosure.

[00063 Figure 4 illustrates a flow diagram of an example of a method for residual data identification according to the present disclosure. [0007] Residua! data includes data instances that do not belong to any recognized category of data instances, identifying residual data instances can include training a classifier with negative data instances and positive data instances. The negative data Instances can include a plurality of data instances in a multi-class training data set that are labeled as belonging to recognized categories. As used herein, a class is intended to be synonymous with a category. The positive data instances can be a plurality of data instances in a first unlabeled data set. Identifying residual data instances can also include applying the classifier to identify residual data instances in a second unlabeled data set.

[0008] A multi-class training data set can include a plurality of data instances that are divided into a number of recognized categories. The plurality of data instances in the number of recognized categories can be considered as negative data instances in training a classifier. The classifier can then be used to identify residual data instances. That is, the classifier can be used to identify data instances that do not belong to the recognized categories.

[00093 * ^{n ti e} present disclosure, reference is made to the accompanying drawings that form a part hereof, and in which is shown by way of illustration how a number of examples of the disclosure can be practiced. These examples are described in sufficient detail to enable those of ordinary skill in the art to practice the examples of this

disclosure, and it is to be understood that other examples can be used and that process, electrical, and/or structural changes can be made without departing from the scope of the present disclosure.

[00103 The figures herein follow a numbering convention in which the first digit corresponds to the drawing figure number and the remaining digits identify an element or component in the drawing. Elements shown in the various figures herein can be added, exchanged, and/or eliminated so as to provide a number of additional examples of the present disclosure. In addition, the proportion and the relative scale of the eiements provided in the figures are intended to itiustrate the examples of the present disclosure, and should not be taken in a limiting sense.

£0011j The specification examples provide a description of the applications and use of the system and method of the present disclosure. Since many examples can be made without departing from the spirit and scope of the system and method of the present disclosure, this

specification sets forth some of the many possible example

configurations and implementations.

[00123 As used herein, "a" or "a number of something can refer to one or more such things. For example, "a number of widgets" can refer to one or more widgets.

[00133 Figure 1 illustrates a block diagram of an example of a computing device 138 according to the present disclosure. The computing device 138 can include a processing resource 139 connected to a memory resource 142, e.g., a computer-readable medium (CRM), machine readable medium (MR ), database, etc. The memory resource 142 can include a number of computing modules. The example of Figur 1 shows a receiving module 143, a labeling module 144, a training module 145, and an application module 146. As used herein, a computing module can include program code, e.g., computer executable instructions, hardware, firmware, and/or !ogic, but includes at least instructions executable by the processing resource 139, e.g., In the form of modules, to perform particu!ar actions, tasks, and functions described in more detail herein in reference to Figure 2A and Figure 2B. The processing resource 139 executing instructions associated with a particular module, e.g., modules 143, 144, 145, and 146, can function as an engine, such as the example engines shown in Figure 3.

[0014] Figure 2A illustrates a diagram of an example of a number of data sets according to the present disclosure. Figure 2B illustrates a diagram of an example of a number of data sets according to the present disclosure. I Figure 2A and Figure 2B, the plurality of data sets can be operated upon by the modules of Figure 1 and the engines of Figure 3.

[00153 Figure 3 illustrates a block diagram of an example of a system 330 for residual data identification according to the present disclosure. The system 330 can perform a number of functions and operations as described in in Figure 2A and Figure 2B, e.g., labeling residual data instances. The system 330 can include a data store 331 connected to a system, e.g., residual data identification system 332. In this example the residual data identification system can include a number of computing engines. The example of Figure 3 shows a receiving engine 333, a training engine 334, a decision threshold engine 335, and a residual data engine 338. As used herein, a computing engine can include hardware, firmware, logic, and/or executable instructions, but includes at least hardware to perform particular actions, tasks and functions described in more detail herein in reference to Figure 2A and Figure 2B.

00 6] The number of engines 333, 334, 335, and 336 shown in Figure 3 and/or the number of modules 143, 144, 145, and 146 shown in Figure 1 can be sub-engines/modules of other engines/modules and/or combined to perform particular actions, tasks, and functions within a particular system and/or computing device. For example, the labeling module 144 and the training module 145 of Figure 1 can be combined into a single module.

[0017] Further, the engines and/or modules described in

connection with Figures 1 and 3 can be located in a single system and/or computing device or reside in separate distinct locations in a distributed computing environment, e.g., cloud computing environment.

Embodiments are not limited to these examples.

[0018] Figure 2A includes a multi-class training data set 206, a first unlabeled data set 208-1 , and a second unlabeled dafa set 208-2. The multi-class training data set 206 and the first unlabeled data set 208-1 can be used to train a classifier that identifies residual data instances.

[0019] The multi-class training data set 206 includes a plurality of data instances, e.g., shown as dots. The multi-class training data set 206 can be received by the receiving module 143 in Figure 1 or the training engine 333 in Figure 3. The labeling module 144 can label the plurality of data instances in the multi-class training data set 206 as belonging to a category 204-1 , a category 204-2, a category 204-3, a category 204-3, a category 204-4, a category 204-5, and/or a category 204-8, e.g., referred to generally as categories 204, In a number of examples, the mu!ti-class training data set 206 can include more or fewer categories than those shown in Figure 2A. In a number of examples, the muits-ciass training data set 206 does not include residua! data and/or data instances that have not been labeled as belonging to a category.

[0020] As used herein, a data instance includes tokens, text, strings, characters, symbols, objects, structures, and/or other

representations. A data instance is a representation of a person, place, thing, problem, computer programming object, time, data, or the like. For example, a data instance can represent a problem that is associated with a product, a web page description, and/or a statistic associated with a website, among othe representations of a data instance. The data instance can describe the problem via text, image, and/or a computer programming object.

[0021] For example, a user of a web site that experiences a problem using the website can fill out a form that includes a textual description and a number of selections thai describe the problem. The form:, the textual description, and/or the number of selections that describe the problem can be examples of data instances. Furthermore, the form, the textual description, and/or the number of selections can be represented as computer programming objects, which can be examples of data instances.

[0022] The data instances can be created manually and/or autonomously. The data instances can be included in a multi-class training data set 206, a first unlabeled data set 208- and/or a second unlabeled data set 208-2.

[0023] The categories 204 describe a correlation between at least two data instances. For example, the category 204-1 can be a type of problem, an organizational structure, and/or a user identifier among other shared commonalities between data instances. For example, the category 204-1 can be a networking problem identifier that describes a specific network problem that is associated with a particular product. Data instances that describe the specific network problem can be iabeled as belonging to the category 204-1. The categories 204 do not include residua! data instances.

[0024] Recognized categories are defined as the categories of the multi-class training data set 206. The data instances in the multi-class training data set 206 can be !abefed as belonging to categories 204 autonomously, e.g., by labeling module 144, and/or by a user. For example, the data instances in the multi-class training data set 206 can be hand-labeled, A user that associates data instances with categories 294 creates a mu!ti-c!ass training data set 206 that has been manually labeled by a user as opposed to being autonomously labeled.

Furthermore, hand-labeled data is data that has had a number of labels confirmed by a user. Predefined classifiers can be applied to divide the data instances into the categories 204. That is, predefined classifiers can be used to autonomously label data instances.

[0025] The first unlabeled data set 208-1 can be received by the receiving module 143 in Figure 1 or the receiving engine 333 in Figure 3. The first unlabeled data set 208-1 includes residua! data instances. Th first unlabeled data set 208-1 may or may not include some data instances that belong in one of the categories 204. It is unknown whether the data instances in the first unlabeled data set 208-1 belong to one of the categories 204 at the time that the first unlabeled data set 208- 1 is received by the receiving module 143 in Figure 1. The first unlabeled data set 208-1 is referred to as unlabeled because the data instances in the first unlabeled data set 208-1 have not been labeled as belonging to the categories 204 and/or labeled as residual data instances as opposed to the data instances in the multi-class training data set 206 that are labeled.

[0026] The first unlabeled data set 208-1 and/or the multi-class training data set 206 can include data instances that are received in a first time period. For example, the first unlabeled data set 208-1 and/or the multi-class training data set 206 can include data instances that describe problems that were encountered with relation to a particular product in a first month. [0027] The second unlabeled data set 208-2 can be received by the receiving module 143 in Figure 1 or the receiving engine 333 in Figure 3, The second uniabeled data set 208-2 includes a plurality of data instances some of which subsequently may be labeled by the trained classifier as residual data. The second unlabeled data set 208-2 includes residua! data instances and/or data instances that belong in one of the categories 204. The second unlabeled data set 208-2 can be received at a second time period. For example, the second unlabeled data set 208-2 can be problems that are reported during a second month.

[0028] The plurality of data instances in the multi-class training data set 206 and the plurality of data instances in the first unlabeled data set 208-1 can be labeled as positive or negative instances by the labeling module 144 in Figure 1. The positive data instances or negative data instances labels applied to the data instances in the multi-class training data set 206 and the first un labeled data set 208-1 can be used in training the classifier by the training module 145 in Figure 1 or the training engine 334 in Figure 3.

[00293 The plurality of data instances in the multi-class training data set 206 can be labeled as negative data instances. Labeling the plurality of data instances in the multi-class training data set 208 as negative data instances can replace the labels that identify the plurality of data Instances in the multi-class training data set 206 as belonging to the categories 204. Negative data instances can represent data instances that are not residua! data instances.

[0030] Th plurality of data instances in the first unlabeled data set 208-1 can be labeled as positive data instances. Positive data instances can represent data instances that the classifier uses to model residual data instances, A classifier models residual data instances by creating a representation of attributes that positive data instances share.

[00313 The data instances in the first unlabeled data set 208-1 can be labeled as positive data instances regardless of whether the data instances are residua! data or whether the data instances belong to the categories 204. That is, the classifier can use data instances that include residua! data and/or non-restduai data to identify residua! data in the second unlabeled data set 208-2. Non-residual data can include data instances that are not residual data, which include data instances that belong to the categories 204,

[00323 ^he training module 145 in Figure 1 or the training engine 334 in Figure 3 can train a classifier using the labeled negative data instances and the labeled positive data instances. The classifier can be a binary classifier, such as a Naive Bayes classifier, decision tree classifier, Support Vector Machine classifier, or any other type of classifier. The classifier, once trained, can identify residual data. That is, the classifier can identify data that does not belong to the categories 204,

[0033] The receiving module 143 in Figure 1 or the receiving engine 333 in Figure 3 can receive the second unlabeled data set 208-2. An application module 146 in Figure 1 or a residual data engine 336 in Figure 3 can apply the classifier to identify residua! data instances in the second unlabeled data set 208-2, The classifier can be applied to the data instances in the second unlabeled data set 208-2 that are provided to the classifier as input. In a number of examples, th classifier can assign a score to each of the data instances. The score can define a level of certainty that a given data instance is residua! data. In a number of examples, the classifier can rank the number of data instances in the second unlabeled data set 208-2 and identify a predetermined number of data instances as residual data. In a number of examples, the classifier can identify whether a given data instance is residual data.

[0034] In a number of examples, a new category can be

suggested and/or created based on the application of a clustering method to the Identified residua! data instances, A known-manner clustering method, such as the K- eans algorithm, can identify

subgroups of the residual data instances that share similarities. A subset of the residual data instances that share the similarities can be included in a new category, A newly created category can represent similarities between data instances and can Include the residual data instances that share the similarities. The residual data instances that belong to the newly created category ar labeled as belonging to the newly created category and are no longer labeled as residual data instances, in a number of examples, the data instances that belong to the new!y created category can be included in the multi-class training data set 208, which can be used to train future classifiers that identify residual data instances.

[00353 a number of examples, the application module 146 in

Figure 1 or the residual data engine 336 in Figure 3 can app!y the classifier to identify residua! data instances in the first unlabeled data set 208-1. The data instances in the first unlabeled data set 208-1 that are not identified as residual data can be removed from the first unlabeled data set 208-1 such that only remaining data instances in the first unlabeled data set 208-1 are treated as residual data instances. That is, data instances that belong to the categories 204 can be removed from the plurality of data instances in the first unlabeled data set 208-1 . Data instances that belong to the categories 204 can be identified by the process of elimination. For example, data instances that are not labeled as residual data by a classifier can be labeled as belonging to the categories 204 without knowing which data instances belong to which of the categories 204. Removing data instances that belong t the categories 204 from the first unlabeled data set 208-1 can further define positive data instances.

[0036] The remaining residual data instances that have not been removed from the first unlabeled data set 208-1 can be labeled as positive data Instances by a labeling module 144 in Figure 1 . A training module 145 in Figure 1 or a training engine 334 in Figure 3 can train a second classifier using the negative data instances and the newly labeled positive data instances. An application module 146 in Figure 1 or a residua! data engine 336 in Figure 3 can apply the second classif ier to identify residual data in the second unlabeled data set 208-2. Applying the second classifier to identify residual data in the second unlabeled data set 208-2 can increase the accuracy in identifying residua! data over the application of the first classifier to identify residual data in the second unlabeled data set 208-2 because the second classifier includes a more accurate model of residual data than the first classifier. The second classifier includes a more accurate model of residual data than the first classifier because the positive data instances used to train the second classifier only include residual data instances while the positive data instances used to train the first classifier can include residual data instances and/or non-residual data instances.

[0O3?3 In a number of examples, a classifier that identifies residua! data instances can be composed of an ensemble of classifiers. The ensemble of classifiers ca identify residual data instances based on a majority vote of the ensemble of classifiers. Each classifier in the ensemble of classifiers can be trained on a subset of labeled positive data instances and labeled negative data instances. The use of an ensemble of classifier to identify residual data instances is further described with respect to Figure 2B.

[0038] Using a classifier to identify residual data instances can be more accurate for identifying residual data instances than using a number of predefined classifiers that identify data instances that belong to the categories 204 and considering a remainder of the data instances to be residual data instances. Each of the predefined classifiers can be trained to identify data instances that belong to one of the categories 204. However, identifying data instances that belong to one of the categories 204 does not identify whether the other data instances are residua! data. For example, a predefined classifier that identifies data instances that belong to the category 204-1 can provide a score that provides a level of certainty that a data instance belongs to the category 204-1 or that the data instance belongs to some other caiegory of multi- class training data set 206. However, the predefined classifier does not identify whether the data instance belongs to the residua! data. Using a classifier that is trained to identify residual data can be more accurate for identifying residual data instances than using a number of predefined classifiers to identify residual data.

[0039] Figure 28 includes a mu!ti-class training data set 206, a first unlabeled data set 208-1 , and a second unlabeled data set 208-2 that are analogous to the mu!ti-c!ass training data set 206, the first unlabeled data set 208-1 , and the second unlabeled data set 208-2 in Figure 2A, respectively. [0040] The receiving engine 333 in Figure 3 or the receiving module 143 in Figure 1 can receive a plurality of data instances in the multi-class training data set. The multi-class training data set 206 can include data instances that belong to a category 204-1 , a category 204-2, a category 204-3, a category 204-4, a category 204-5, and a category 204-6, e.g., referred to generally as categories 204. The categories 204 are analogous to the categories 204 in Figure 2 A.

[0041] The multi-class training data set 206 also includes a number of sections that further divide the data instances. For example, the data instances in the multi-class training data set 206 can be divided into a section 210-1 , a section 210-2, a section 210-3, a section 210-4, a section 210-5, a section 210-6, a section 210-7, a section 210-8, a section 210-9, a section 210-10, a section 210-1 1 , a section 210-12, a section 210-13, a section 210-14, a section 210-15, a section 210-16, a section 210-17, and a section 210-18.

[0042] The receiving engine 333 in Figure 3 or the receiving module 143 in Figure 1 can receive a plurality of data instances in the first unlabeled data set 208-1. The data instances in the first unlabeled data set 208-1 can be divided into a section 210-19, a section 210-20, and a section 210-21 . The sections in the multi-class training data set 206 and the sections in the first unlabeled data set 208-1 are referred to generally as sections 210, The multi-class training data set 206 and/or the first unlabeled data set 208-1 can be divided into more or fewer sections than those described herein,

[0043] As used herein, a section can include a subset of the data instances that belong to a category. Sections are used to divide data instances within the categories 204. Sections can be used to a plurality of classifiers with different data instances. For example, the section 210- 1 can be a first subset of the data instances in category 204-1 , the section 210-7 can be a second subset of the data instances in category 204-1 , and the section 210-13 can be a third subset of the data instances in category 204-1.

[0044] The training engine 334 in Figure 3 or the training module 145 in Figure 1 can train a plurality of classifiers to identify residual data instances. For exampie, the training engine 334 in Figure 3 or the training module 145 in Figure 1 can train a first classifier, a second classifier, and a third classifier to identify residual data instances. More or fewer classifiers can be trained to identify residua! data instances. The first classifier, the second classifier, and the third classifier referred to in Figure 2B are different than the first classifier and the second classifier referred to in Figure 2A because the first classifier, the second classifier, and the third classifier referred to in Figure 2B can collectively identify residua! data instances while the first classifier and the second classifier referred to in Figure 2A independently identify residual data instances. That is, a first classifier or a second classifier in Figure 2A can consist of a first classifier, a second classifier, and a third classifier as described in Figure 2B.

£0045] The first classifier, the second classifier, and/or the third classifier can be independent from each other. The data instances used to train the first classifier can be different than the data instances used to train the second classifier and/or th third classifier. Sn a number of examples, the data instances used to train the first classifier can be used to train the second ciassifier and/or the third classifier.

[0046] Each of the classifiers that identify residual data instances can be trained using one or more of the plurality of sections, e.g., section 210-1 through section 210-18, of the plurality of data instances in the multi-class training data set 206 as negative data instances. Each of the classifiers that identify residual data instances can be trained using one or more of first sections, e.g., section 210-19 through section 210-21, of the plurality of data instances in the first unlabeled data set 208-1 as positive data instances.

[0047] For exampie, an n-fold cross validation method can be used to train the plurality of classifiers. In the examples given in Figure 2B, 3-fold cross validation is used to train the three classifiers using three different groupings of section 210-1 through section 210-19, and three different groupings of section 210-19 through section 210-21. However, for example, 10-fold cross validation can be used among other variations of n-fold cross validation. The letter "n ^B in n-fo!d cross validation represents the number of classifiers that are trained to identify residuai data and/or the number of sets of data that are used to train the number of classifiers. The data instances in section 210-1 through section 210- 12 can be used as negative data instances to train a first classifier that identifies residua! data instances. The data instances in section 210-7 though section 210-18 can be used as negative data instances to train a second classifier that identifies residual data instances. The data instances in section 210-13 through section 210-18 and section 210-1 through section 210-6 can be used as negative data instances to train a third classifier that identifies residual data instances. The data instances in section 210-19 and section 2 0-20 can be used as positive data instance to train the first classifier. The data instances in section 210-20 and section 210-21 can be used as positive data instances to train the second classifier. The data instances in section 210-21 and section 210- 19 can be used as positive data instances to train the third classifier.

[0048] A decision threshold engine 335 in Figure 3 can set a decision threshold for each of the plurality of classifiers based on one of the plurality of second sections, e.g., section 210-19 through section 210- 21 , of the plurality of data instances in the first unlabeled data set 208-1. The plurality of second sections can include the same sections, e.g., section 210-19 through section 210-21 , as the plurality of first sections because any given classifier only uses data instances in a portion of the available sections as positive data instances. Data instances in the remaining portion of the available sections are used to set the decision threshold,

[0049] Fo example, a first grouping of the plurality of first sections can include the section 210-19 and section 210-20. The first grouping of the plurality of first sections can be used to train the first classifier. The remaining section, e.g., section 210-21 can be included in the plurality of second sections. A second grouping of the plurality of first sections can include section 210-20 and section 210-21 , The second grouping of the plurality of first sections can be used to train the second classifier. The remaining section, e.g., section 210-19, can be included in the plurality of second sections. A third grouping of the plurality of first sections can be included in section 210-19 and section 210-2 The third grouping of the plurality of first sections can be used to train the third classifier. The remaining section, e.g., section 210-20, can be included in the plurality of second sections. That is, the plurality of first sections can include the section 210-19 and the section 210-20, the section 210-20 and the section 210-21 , and the section 210-19 and the section 210-21. The piurality of second sections can inciude the section 210-19, the section 210-20, and the section 210-21.

[0O§03 Data instances in the section 210-21 can be used to set a first decision threshold for a first classifier if data instances in the section 210-19 and the section 210-20 are used as positive data instances in training the first classifier. Data instances in the section 210-19 can be used to set a second decision threshold for a second classifier if data instances in the section 210-20 and the section 210-21 are used as positive data instances in training the second classifier. Data instances in the section 210-20 can be used to set a third decision threshold for a third classifier if data instances in the section 210-19 and the section 210-21 are used as positive data instances in training the third classifiers.

[0051j| A decision threshold can be set such that a predefined percentage of data Instances in an associated section from the plurality of second sections are identified by an associated classifier as non- residual data instances. For example, given that the data instances in the section 210-19 and the section 210-20 are used as positive data instances, then the data instances in the section 210-21 can be used to set the decision threshold for a given classifier. The given classifier can give a score to a data instance that can be used io determine whether the data instance is residual data. The plurality of data instances in the section 210-13 through the section 210-18 can be ranked based on a score that is given by the given classifier to each of the plurality of data instances. A decision threshold can be a number that coincides with the score such that a predefined percentage of the plurality of scores are below the decision threshold. For example, given that there are 100 data instances in the section 210-13 through the section 210-18, that each of the 100 data instances are given a score, and thai the predefined percentage is set at 98 percent, then a decision threshold can be set such that 98 percent of the scores, and as a result 98 percent of the associated data instances, are below the decision threshold. The data instances that have an associated score that falls below the decision threshold can be identified as non-residua! data instances b the given classifier. The data instances that have an associated score that fails above the decision threshold can be identified as residua! data instances by the given classifier.

£00523 In a number of examples, a bagging method can be used to train the plurality of classifiers. A bagging method can use a randomly selected p!urality of data instances from each section, e.g., section 210-1 through section 210-18, in the multi-class training data set 206 as negative data instances. The bagging method can use data instances in a randomly selected plurality of section, e.g., section 210-19 through section 210-21 , in the first uniabe!ed data set 208-1 as positive data instances. A decision threshold can be set such as defined above using the unse!ected data instances from the muiti-class training data set 206 given classifiers that are trained using the bagging method.

[0O53j| The residua! data engine 336 in Figure 3 and the

application module 148 in Figure 1 can identify data instances as residua! data when a majority of the plurality of classifiers identif the data instances as residua! data. For example, given that three classifiers are trained using the examples given in Figure 2B _: then each of the three classifiers can identify each of the plurality of data instances in the second unlabeled data set 208-2 as residual data or non-residual data.

[0054] Fo example, a first classifier can identify a data instance as residual data, a second classifier can identify the data instances as residua! data, and a third classifier can identify the data instance as non- residua! data. Each of the identifications given by plurality of classifiers can be said to be a vote. For example, the first classifier can vote that the data Instance Is residua! data, the second classifier can vote that the data instance is residual data, and the third classifier can vote that the data instance is non-residual data. A majority of the votes, and/or a majority of the identifiers given by the plurality of classifiers can be used to label the data instance as residual data. The classifiers can be used to collectively label data instances using a different combination of the classifiers and/or different measures given by the classifiers,

[OQSSj Figure 4 illustrates a flow diagram of an example of a method for residual data identification according to the present disclosure. At 450, a plurality of data instances in a second unlabeled data set can be received. The second unlabeled data set can be second as compared to a first unlabeled data set The use of first and second with relation to the unlabeled data sets does not imply order but is used to conform to naming conventions used in Figures 2A and 2B.

[0056] At 451 , the plurality of data instances in the second unlabeled data set can be ranked. The ranking can be based on a score assigned by a classifier to each of the plurality of data instances in the second unlabeled data set.

[0057] At 452, each of the plurality of data instances in the second unlabeled data set can be compared to at least one of a plurality of characteristics which distinguishes negative data instances that include a plurality of data instances in a multi-class training data set and positive data instances that include a plurality of data instances in the first unlabeled data set. The plurality of data instances in the second unlabeled dat set can be compared to the characteristics of the negative data instances and the positive data instances to score each of the plurality of data instances in the second unlabeled data set. A score can describe a similarity between a data instances and the positive data instances and/or the negative data instances. For example, a high score can indicate that a data instance shares more similarities with the positive data instances than the negative data instances. A comparison can be a means of producing the score using a trained model and a data instance,

[0058] In a number of examples, a classifier can include a model of the positive data instance and the negative data instances. The training module 145 in Figure 1 and the training engine 334 in Figure 3 can train the classifier by creating a model which can be referred to herein as a trained model. A model can describe a plurality of characteristics associated with the negative data instances and a plurality of characteristics associated with the positive data instances.

[0059] At 453, a number of the ranked plurality of data instances in the second unlabeled data set can be identified as residual data based o a threshold value applies to the ranked plurality of data instances. A threshold value can be used to determine which of the ranked plurality of data instances are identified as residual data and/or non-residual data. For example, if a threshold value is .75, then data instances with a score equal to and/or higher than .75 can be Identified at residual data. In a number of examples, a threshold value can define a number of the plurality of data instances that are residual data. For example, if a residual value is 10, then the 10 data instances with the highest score can be identified as residual data.

£0060] The threshold value can be pre-defined and/or or set by a quantification technique. A pre-defined threshold value is a threshold value that is hand selected and/or a threshold value that does not change. For example, a pre-defined threshold value can b selected during the training of a classifier, before the training of the classifier, and/or after the training of the classifier by a human user.

[0061] A quantification technique can provide a number of expected data instances that have a potential for being identified as residual data instances. A quantification method can predict a number of data instances that should be identified as residua! data and/or a percentage of the data instances in the second unlabeled data set that should be identified as residual data. For example, a quantification method can predict that a second unlabeled data set includes 400 residual data instances. A threshold value can then be set so that 400 data instances in the second unlabeled data set are selected as residual data. Similarly, if a quantification method predicts that 5 percent of the data instances in the second unlabeled data set should be identified at residual data, then a threshold value can be set such that 5 percent of the ranked data instances in the second unlabeled data set are identified as residual data. A quantification techniqu can use th multi-class training data set and/or the first and second unlabeled data sets to create a prediction. The prediction can be based on the number of residual data instances observed in the first unlabeled data set as compared to the non-residual data instances observed in the multi-class training data set and/or the first unlabeled training data set.

[0062] The threshold value is selected to comprise a threshold level that satisfies at least one condition. The possible conditions include selecting the threshold value to substantially maximize a difference between the true positive rate (TPR) and the false positive rate (FPR) for the classifier, so that the false negative rate (FNR) is substantially equal to the FPR for the classifier, so that the FPR is substantially equal to a fixed target value, so that the TPR is substantially equal to a fixed target value, so that the difference between a raw count and the product of the FPR and the TPR is substantially maximized, so that the difference between the TPR and the FPR is greater than a fixed target value, so that the difference between the raw count and the FPR multiplied by the number of data instances in the target set is greater than a fixed target value, and based on a utility and one or more measures of behavior. As used herein, substantially indicates within a predetermined level of variation. For example, substantially maximizing a difference includes maximizing a difference beyond a predetermined difference value.

Furthermore, substantially equal includes two different values that differ by less than a predetermined value.

[00633 ^{n a} number of examples, the selected threshold level worsens the ability of the classifier to accurately classify the data instances. However, the accuracy in the overall count estimates of the data instances classified into a particular category is improved. In addition, the classifier employs the selected threshold value, along with various other criteria to determine whether the data instances are residua! data or non-residual data. Moreover, one or both of a count and an adjusted count of the number of data instances that are residual data are computed,

[0064] In a number of examples, multiple intermediate counts are computed using a plurality of alternative threshold values. Some of the intermediate counts are removed from consideration and the median, average, or both, of the remaining intermediate counts are determined, The median, average, or both of the remaining intermediate counts are then used to calculate an adjusted count. Using the quantification technique describe herein, the data instances in the second unlabeled data set can be identified as residua! data or non-residual data.