PROCESSES TO CORRECT FOR BIASES AND INACCURACIES

CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] The present invention claims priority to U.S. Provisional Patent Application Serial No. 62/595,474 entitled "Processes To Evaluate Intrinsic Quality" filed December 6, 2017. The disclosure of U.S. Provisional Patent Application Serial No. 62/595,474 is herein incorporated by reference in its entirety.

FIELD OF THE INVENTION

[0002] The present invention generally relates to data analytics, including processes for correcting for biases and inacuracies in data sets, and more specifically relates to determining accuracies and biases of data sources and qualities of items.

BACKGROUND

[0003] Most goods and services that humans consume are evaluated and rated. Prominent examples include (but are not limited to) films, theater, art, books, games, wines, restaurants, bars, clubs, stocks, professional services, transportation services, hotels, universities, teachers, and most consumer products. The internet and various platforms have led to enormous growth in the number of items that are evaluated and the number of people generating online rating information. Selling platforms on the web most often report previous consumers ratings. Numerous other websites collect evaluations from distributed sources and report them to the public. These ratings can come from experts (movie critique ratings) or from users (e.g. Yelp). Importantly, such ratings can provide dramatic increases in market efficiency. In fact, an important innovation that has accompanied the "digital economy" are the ratings that are available about almost everything.

SUMMARY OF THE INVENTION

[0004] Systems and methods are described for correcting biases and inaccuracies in data sets. In an embodiment: A compilation of quality indicators of a set of items is received using a computer system. Each item has been provided a quality indicator by a set of data sources. The set of items is at least two items. The set of data sources is at least two data sources. A first data source and a second data source, of the set of data sources, have each provided a quality indicator of a first item and a second item, of the set of items, an initial estimate of an error and a bias of each data source in the set of data sources is determined using the computer system. An initial estimate of a quality of each item is determined using the computer system. The estimate of the quality of each item, of the set of items are centered, using the computer system, at a current estimate of the mean quality of all items in the set of items. The estimates of the quality of each item, the error of each data source, and the bias of each data source are solved using the computer systems. Furthermore, the centering of the estimate of the quality of each item at a current estimate of the mean quality of all items and the solving of the estimates of the quality of each item, the error of each data source, and the bias of each data source are iteratively repeated using the computer system, until the estimates converge into a solution that provides a final quality of each item, a final accuracy of each data source, and a final bias of each data source.

[0005] In a further embodiment, the quality of each item is solved at each iteration with a formula:

wherein is the quality q of an item i at iteration t, <¾· is the quality indicator of item i by a data source is the bias 6 of a data source j at iteration t, is the error σ| of a data source j at iteration i, and Q\ is the overall mean quality in iteration t.

[0006] In another embodiment, the error of each data source is solved at each iteration with a formula:

wherein is the error aj of a data source j at iteration ί, gij is the quality indicator

of item i by a data source j, qj is the quality q of an item i at iteration t, &*· is the bias 6 of a data source j at iteration t, and rij is the total number n of data sources j.

[0007] In a still further embodiment, the bias of each data source is solved at each iteration with a formula:

wherein 6^· is the bias 6 of a data source j at iteration t, gij is the quality indicator of item i by a data source j, q\ is the quality q of an item i at iteration t, and rij is the total number n of data sources j.

[0008] In still another embodiment, the estimates of each itemas quality are centered at a current estimate of the mean quality of all items with an equation:

wherein <¾j is the quality indicator of item i by a data source j, n is the total number of items i, rij is the total number of data sources j, rrii is the number quality indicators for each item and is the best current estimate of the overall average true quality through

iteration

[0009] In a yet further embodiment, the initial estimate of each data sourceas error is an arbitrary positive number.

[0010] In yet another embodiment, the initial estimate of each data source's bias is calculated using a formula:

wherein is the quality indicator of item i by a data source n is the total number of items i is the total number of data sources j and is the number quality indicators for

each item i.

[0011] In a further embodiment again, the initial estimate of each item's quality is calculated using formulas:

and

wherein gij is the quality indicator of item i by a data source j, n is the total number of items i, rij is the total number of data sources j, mi is the number quality indicators for each item i, and is the overall mean quality in iteration t.

[0012] In another embodiment again, the first item is priced based upon the final quality of the first item.

[0013] In a further additional embodiment, the first and the second items are displayed in an order based upon the final qualities of the first and the second items. [0014] In another additional embodiment, the first and the second items are displayed on an online marketplace.

[0015] In a still yet further embodiment, the first item is displayed when the final quality of the first item exceeds a threshold.

[0016] In still yet another embodiment, the first item is displayed on an online marketplace.

[0017] In still yet another embodiment, the first item is imported when the final quality of the first item exceeds a threshold.

[0018] In still yet another embodiment, a regulatory standard is set based at least upon the final quality of the first item.

[0019] In still yet another embodiment, the first item is a consumer product.

[0020] In still yet another embodiment, the consumer product is an electronic, grocery, clothing, or vehicle.

[0021] In still yet another embodiment, the consumer product is wine.

[0022] In still yet another embodiment, the first item is a professional service.

[0023] In still yet another embodiment, the professional service is a medical service, a contractor service, a legal service, or a brokerage service.

[0024] In still yet another embodiment, the first item is an entertainment program.

[0025] In still yet another embodiment, the entertainment program is cinema, theater, television, online streaming, music, or literature.

[0026] In still yet another embodiment, the first item is an investment security.

[0027] In still yet another embodiment, the first item is a food and beverage establishment.

[0028] In still yet another embodiment, the food and beverage establishment is a restaurant, a bar, a club, a winery, a brewery, or a catering establishment.

[0029] In still yet another embodiment, the first item is an educational service.

[0030] In still yet another embodiment, the educational service is a university, a college, a teacher, or a test preparation course.

[0031] In still yet another embodiment, the first item is a transportation and travel service.

[0032] In still yet another embodiment, the educational service is a hotel, an airline, a train, a rental car service, or a ridesharing service.

[0033] In still yet another embodiment, the first item is a game.

[0034] In still yet another embodiment, the first item is a sport team.

[0035] In still yet another embodiment, a fraudulent quality indicator within the compilation of quality indicators is identified using the computer system that utilizes a distribution of quality indicators of at least one data source of the set of data sources. The fraudulent quality indicator is removed, using the computer system, from the compilation of quality indicators prior to solving the final quality of each item in the set of items, the final accuracy of each data source of the set of data sources, and the final bias of each data source of the set of data sources.

[0036] In still yet another embodiment, the data source is a rater and the quality indicator is a rating.

BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Fig. 1 is a flow chart illustrating a process to utilize a determined intrinsic quality in accordance with an embodiment of the invention.

[0038] Fig. 2 is a flow chart illustrating a process to normalize scaled ratings in accordance with an embodiment of the invention.

[0039] Fig. 3 is a flow chart illustrating a process to determine a true intrinsic quality of an item in accordance with an embodiment of the invention.

[0040] Fig. 4 is a conceptual diagram of a computer system configured to determine a true intrinsic quality of an item in accordance with various embodiments of the invention.

[0041] Fig. 5 provides charts of note and wine distribution across vintage years, utilized in accordance with various embodiments of the invention.

[0042] Fig. 6 provides charts of kernel notes density plots of a number of wine expert raters, utilized in accordance with various embodiments of the invention.

[0043] Fig. 7 provides charts of normalized ratings distributions of two wine expert raters, generated and utilized in accordance with various embodiments of the invention.

[0044] Fig. 8 provides charts of convergence of estimated quality of items, generated in accordance with various embodiments of the invention.

[0045] Fig. 9 provides charts of convergence of estimated expert biases and inaccuracies, generated in accordance with various embodiments of the invention.

[0046] Fig. 10 A provides charts of normalized expert biases with estimated quality, generated in accordance with various embodiments of the invention.

[0047] Fig. 10B provides charts of normalized expert accuracies and correlation coefficients of expert rates with estimated quality, generated in accordance with various embodiments of the invention.

[0048] Fig. 11 provides a chart of the correlation of expert accuracies with predicted expert accuracies, generated in accordance with various embodiments of the invention.

[0049] Fig. 12 provides charts of estimated qualities of wine, generated in accordance with various embodiments of the invention.

[0050] Fig. 13 provides a chart of rescaling estimated quality onto a scale utilized by an expert wine rater, utilized in accordance with various embodiments of the invention.

[0051] Fig. 14 provides charts of distribution wine prices in three markets, utilized in accordance with various embodiments of the invention.

[0052] Fig. 15 provides a chart showing the correlation of expert wine rater accuracy with rater rating/price correlation, utilized in accordance with various embodiments of the invention.

[0053] Figs. 16A and 16B provide charts of normalized expert accuracies and biases with estimated quality for the left and right banks, generated in accordance with various embodiments of the invention.

[0054] Figs. 17 and 18 provide charts of difference of expert accuracies and biases with estimated quality, accounting for the left and right banks, generated in accordance with various embodiments of the invention.

DETAILED DESCRIPTION

[0055] Turning now to the drawings and data, a number of embodiments of processes to correct for biases and innacuracies of subjective data sources are provided. Signal processing techniques that involve evaluating and estimating intrinsic qualities of ratable items and accuracies and biases of raters of these items in accordance with various embodiments of the invention are illustrated. In several embodiments, processes encompass computational and statistical evaluation of ratings. In contrast to commonly practiced methods to evaluate ratings that utilize simple aggregation techniques, which necessarily include rater biases and inaccuracies, many embodiments of the invention use processes to discover biases and inaccuracies and correct for them to achieve a compilation of ratings that reach intrinsic qualities of the rated items. Once estimates of true intrinsic qualities of items are revealed, in accordance with several embodiments, intrinsic qualities can be used in further downstream applications, including (but not limited to) price valuation, product placement, product import and export, regulatory violations, and marketing. And in some embodiments, rater biases and inaccuracies are utilized to discover fake raters and/or fake ratings that are in- congruent with a rater's history.

[0056] There are many challenges with production of ratings as individual raters may have significant biases and may vary widely in their accuracy in evaluating the quality of a product. Simply averaging ratings can provide biased results, especially for items that have fewer ratings. For example, if some people only rate items that they have extreme experiences with, they may consistently err in terms of making excessively extreme ratings. Also, ratings of people who are very careful and discerning are mixed in with others who are careless and frivolous. To make the most of such rating information, in accordance with a number of embodiments, systems for processing such ratings take into account reviewers' histories of past ratings to evaluate their biases and accuracies. By undoing biases and putting more weight on the most accurate reviewers, well-processed ratings can result in significant improvements in the quality of an aggregate rating.

[0057] In accordance with a number of embodiments, systems for processing ratings simultaneously estimate intrinsic qualities of ratable items, the accuracies of raters, and biases of each particular rater. In several embodiments, estimations of qualities, accuracies, and biases are performed on an iterative basis until converged. Convergence of estimates, in accordance with many embodiments, reveals a true intrinsic quality of an item. In numerous embodiments, intrinsic quality is used to evaluate an item's monetary value. In some embodiments, an item's monetary value is determined before the item enters a commercial market. And in various embodiments, true intrinsic qualities determined by processes described herein are better predictors of item prices than average rater scores.

[0058] Many embodiments are also directed to revealing rater accuracy and bias. Accordingly, embodiments are directed to evaluating raters on their accuracies and/or biases. In several embodiments, processes are used that provide some immunity to manipulation of ratings and/or selection biases. In some embodiments, the processes correct the accuracy of raters to converge on an intrinsic quality of an item. In a number of embodiments, the processes work around biases of raters such that their biases are at least partially mitigated, if not fully eliminated in determined intrinsic qualities.

[0059] Various embodiments are directed to revealing when a rater provides a fraudulent rating. Utilizing a ratings and reviews, anomalous and/or outlying ratings are detected in accordance with several embodiments. A number of embodiments detect when an item and/or a rater has a pattern indicative of fraudulent reviews (e.g., when high reviews are bribed). In some embodiments, reviews deemed fraudulent are flagged and/or removed from item quality analysis.

[0060] In a number of embodiments, scales of the various raters are adjusted to normalize the ratings of items. For example, one rater may use a scale of 1 to 20 and another rater may use a scale of 1 to 100. According to a number of embodiments, the ratings are normalized to each other so that each rating is on a common and commensurable scale. In many embodiments, normalized ratings are used in a number of processes to estimate intrinsic qualities of ratable items.

[0061] In various embodiments, the intrinsic quality of a ratable item is determined. In some embodiments, a ratable item is any item that is rated by a group of individuals. Ratable items, in accordance with several embodiments, include (but are not limited to) consumer products, professional services, food and beverage establishments, entertainment programs, games, sport teams, educational services, transportation and travel services and investment securities.

[0062] In a number embodiments, a consumer product is an item available for purchase by a consumer. Consumer products may include (but are not limited to) electronics, groceries, clothing, vehicles, and other retail. In various embodiments, a professional service is an action performed by an individual for another individual, typically for a fee. Professional services may include (but are not limited to) medical services (e.g., doctors), contractor services, legal services (e.g., attorneys), and brokerage services. In some embodiments, a food and beverage establishment is one that provides a food and/or beverage service, often including table and/or bar service. Food and beverage establishment may include (but are not limited to) restaurants, bars, clubs, wineries, breweries, and catering. Likewise, in numerous embodiments, an entertainment program is a product for consumer enjoyment. Entertainment programs may include (but are not limited to) cinema, theater, television, online streaming, music, literature. In several embodiments, educational services are services meant to provide a learning experience. Educational services may include (but are not limited to) universities, colleges, teachers, and test preparation courses. Transportation and travel services are services that provide means to travel. Transportation and travel services may include (but are not limited to) hotels, airlines, trains, rental cars, and ridesharing.

[0063] Numerous exemplary embodiments described herein incorporate various processes described herein to determine the intrinsic qualities of wines. In many exemplary embodiments, ratings from experts in the field of wine tasting are utilized to determine a true intrinsic quality of various wines and vintages. In the wine industry it is common for experts to taste and rate wine before it is bottled resulting in ratings known in the industry as an "en primeur" rating. Accordingly, in various exemplary embodiments, "en primeur" ratings are incorporated into processes described within to obtain an intrinsic quality of a wine before it is bottled. In a number of exemplary embodiments, an intrinsic quality of a wine is determined before the wine enters into consumer markets. In several exemplary embodiments, an intrinsic quality of a wine as determined by processes described within is used to determine a future market value of the wine. It should be noted that, although applications to wine industry are described, the various embodiments as detailed within can be implemented in a number of applications, including (but not limited to) applications to films, theater, art, books, restaurants, investment securities, and most consumer products.

Definitions and Notation

[0064] In order to easily understand the various embodiments described within, the following notations are used. It should be understood, however, that these notations are merely provided to help guide a reader's comprehension of the various described embodiments and processes. These notations are not meant to be limiting in any way. For example, a set of items is denoted as N, however any notation could be used to describe a set of items.

[0065] A set N of items i = 1, ...,n is to be rated.

[0066] A set M of raters j = 1, ...,ra each rate a specific subset of the items M

[0067] The ratings are listed in the n x m matrix g with the G IR being j's rating of item i, and with g _% j = . (missing information) indicating that j did not rate item i.

[0068] Let Uj be the indicator variable that is 1 if rater j rated item i, and 0 otherwise (so it is the indicator that <¾ _j φ .).

[0069] Let be the number of the number of ratings of item i and r

the number of ratings by expert j.

Overview of Bias and Error Correction Processes

[0070] Provided in Fig. 1 is an overview process of correcting for biases and inaccuracies to determine a quality of an item and then utilizing the determined quality to perform an application. Accordingly, the process begins with correcting (101) for biases and inaccuracies to determine a quality of an item. In numerous embodiments, the item is a ratable item and the data sources are raters that rate the item. In various embodiments, an intrinsic quality is determined, which is a quality as determined by collection of individual data sources, correcting for each data source's bias and inaccuracy. Accordingly, a true intrinsic quality should be free of inaccuracies and subjective biases.

[0071] As described herein, various embodiments utilize a matrix of data sources and items with enough overlap such that biases and inaccuracies of data sources are determined. In several embodiments, at least two data sources, each providing an indicator of quality for at least two items, is necessary to determine an intrinsic quality of each of the two items. It should be understood, however, that more data sources, each providing quality indicators for multiple items such that a history of each data source can be established to inform of biases and inaccuracies, will produce a more accurate intrinsic quality. In some embodiments, iterative computations of each item's quality and each data source's accuracy and bias are solved until a convergence is reached. Accordingly, various embodiments incorporate Bayesian updating, minimizing some moment function, minimizing the squared errors, or a combination of thereof. In some embodiments, a Generalized Method of Moments is used to solve an items final quality, and the final biases and inaccuracies of each data source.

[0072] Once a quality of an item is determined, in accordance with various embodiments, the quality is utilized (103) to perform an action on the item. In some embodiments, a quality is used for price valuation, product placement, product import and export, regulatory violations, and marketing.

[0073] In a number of embodiments, an item's quality is used to set a price valuation. For example, in some embodiments, as the higher the quality of an item, the higher the item is valued and priced. In several embodiments, an item's quality is utilized to place a product. For example, various embodiments will sort a product on an online marketplace (e.g., Amazon.com) such that items are displayed in order of their quality. In various embodiments, only items having a quality equal to and/or above a particular quality threshold are displayed. Likewise, various embodiments utilize quality to determine whether an item is imported/exported when the item has a quality equal to and/or above a particular quality threshold. And in several embodiments, a quality of an item is used to set regulatory standards. Further, embodiments utilize quality to set up appropriate products or services prices or to decide to commercialize them or not. Various embodiments compare a set of products' quality to define appropriate products or services prices in a sales period.

[0074] In several embodiments, the biases and accuracies of data sources are used to inform about each data source. The estimated error (inverse accuracy) of data sources, calculated as the average squared difference between the estimated item quality and the quality indicator provided by the data source (corrected fir bias), is used in some embodiments to appreciate the data source's reliability. Some embodiments calculate the accuracy of data source for specific types of products to determine on which submarket the data source's knowledge is more dense. When the data sources are raters, various embodiments compare a rater's reliability to prioritize the rater's ratings or the comments the rater provides, or to target the rater in a commercial or incentivizing policy.

[0075] Various embodiments are directed to determining whether a rating is fraudulent. There are many instances in which raters have been reported to be paid or bribed to provide a certain rating, from rating games to providing online reviews of restaurants. In some cases, a product might even create a fake reviewer just to review its product. More generally, this involves bribing well-established and visible reviewers to deliberately give a product a high rating.

[0076] There are a few cases in which techniques described herein can identify whether there are fraudulent reviews. In some embodiments, fraud can be detected when many raters rate a particular item, and a nontrivial fraction but not all of them are bribed. This scenario results in a pattern in which the distribution of ratings does not follow the usual random pattern around the raters' biased points obtained, but instead has an extra mode at a high level with a statistically rare and identifiable number of ratings that deviate from their mean. In more embodiments, fraud can be detected when a given reviewer is bribed on a non-trivial fraction of items. In this scenario, a rater has an abnormally high number of ratings that are outliers, as detected utilizing the rater's bias and accuracy and the true quality of the items obtained. Accordingly, in a number of embodiments, when a statistically rare and identifiable number of ratings that are outliers, these ratings are flagged and/or removed from quality analysis for being fraudulent.

Rating Acquisition and Processing

[0077] A conceptual illustration of a process to normalize numerically scaled ratings utilizing computer systems in accordance with an embodiment of the invention is provided in Fig. 2. In a number of embodiments, multiple ratings of multiple ratable items are obtained (201) and compiled. An item, in accordance with several embodiments, is any item rated by a group of raters. In many embodiments, items include (but are not limited to) consumer products, professional services, restaurants, entertainment programs (e.g., movies, theater, music, books), and investment securities.

[0078] A rater, in accordance with a number of embodiments, is any individual that provides a numerical rating, ranking, or a narrative review on an item. In more embodiments, a rater provides a qualitative narrative review that may be converted into a quantitative numerical rating. Accordingly, in some embodiments, raters are consumers that provide feedback on items previously purchased or used. In more embodiments, raters are experts in an industry that rate, rank, and/or review various items professionally.

[0079] In more embodiments, a rater j independently estimates an item's i true quality q¾. In some embodiments, rater j may have systematic bias bj, consistently over or under rating the true quality. In further embodiments, raters introduce error into their ratings of items. In even more embodiments, ratings consider an item's true quality, systemic bias, and error. In some embodiments, a rater's observed rating is defined by the equation:

where is for the same j, and independent across j, and uncorrected with the

¾. More embodiments are also directed to defining a rater's accuracy as the inversed square of her error:

[0080] Ratable items to be assessed, in accordance with some embodiments, is defined within a category. In several embodiments, a compilation of rating category is defined by the item to be assessed (e.g., movies). In many embodiments, a category is defined by the raters (e.g., Yelp users). It should be noted, however, some processes do not necessitate a categorical definition of the compilation. In various embodiments, categorical definitions are beneficial to associate a group of items and or raters for comparison. [0081] In numerous embodiments, obtained ratings are numerically scaled. In several embodiments, numerical rankings are obtained and utilized as ratings. In many embodiments, narrative reviews are obtained and converted into scaled numerical ratings. Accordingly, in various embodiments, a quantitative rating value reflects raters' qualitative opinion of an item. The scale of the rating, in accordance with many more embodiments, is any numerical scale, so long that numerical values correspond with quality of items as determined by a rater. In some embodiments, ratings are scaled from zero to a hundred. In a number of embodiments, ratings are scaled from zero to twenty. In numerous embodiments, ratings are scaled from one to five. In several embodiments, a higher numerical score corresponds with a higher quality. In a multitude of embodiments, lower numerical scores correspond with a higher quality (e.g., ratings based on rankings of items).

[0082] Obtained ratings are normalized (203), in accordance with various embodiments, such that each rating is on a commensurable scale when compared to the collection of rankings. Often, ratings can be collected having different scales and/or different distributions. For example, two raters may each use a zero to one hundred scale, but one rater may typically only rate items between seventy and one hundred with an average of ninety and the other rater may typically rate items fifty to a hundred with an average of eighty. Despite having the same theoretical scale, the differences of distribution result in different scales between the two raters, and thus should be normalized to a commensurable scale. Accordingly, a number of embodiments are directed to aligning some order statistics of the distributions and translating them to a common scale.

[0083] In accordance with several embodiments, obtained ratings are rescaled to a scale defined by a user, and may be dependent on the application. In many embodiments, a user defined scale of the collection of ratings does not matter, as long as each rating of the collection of ratings are rescaled to the same commensurable scale and distribution. In some embodiments, each obtained rating is rescaled to a scale of zero to one hundred. In a number embodiments, each obtained rating's scored average is reset to fifty when a scale of zero to one hundred is used. In numerous embodiments, each obtained rating is linearized.

[0084] Many embodiments are also directed to trimming each set of ratings of a rater. In certain cases, tails of a rater's compilation of ratings may be noisy and or long. Accordingly, in various embodiments, a certain amount of each rater's compilation of rating is removed from further analysis. In some embodiments, the removed ratings are a certain amount of at least one tail. In many embodiments, a certain amount of the lower tail of each rater's compilation of ratings is removed. In particular embodiments, the lowest five percent of each rater's compilation of ratings is removed.

[0085] In a number of embodiments, raw ratings of each rater are rescaled using the equa- tion:

where G denotes the raw score as described by the rater, S defines the linear scale, pi denotes the lower bound of ratings utilized, and p _u defines the upper bound of ratings used.

[0086] In accordance with several embodiments, normalized scaled ratings are stored and/or reported (205). In further embodiments, normalized scaled ratings may be used in many further downstream applications, including (but not limited to) further statistical analysis on the ratings.

[0087] While a specific example of a process for normalizing a collection of scaled ratings is described above, one of ordinary skill in the art can appreciate that various steps of the process can be performed in different orders and that certain steps may be optional according to some embodiments of the invention. As such, it should be clear that the various steps of the process could be used as appropriate to the requirements of specific applications. Furthermore, any of a variety of processes for normalizing a collection of scaled ratings appropriate to the requirements of a given application can be utilized in accordance with various embodiments of the invention.

Intrinsic Quality of an Item

[0088] In accordance with numerous embodiments of the invention, intrinsic qualities of items are determined utilizing ratings. Fiuihermore, in accordance with several embodiments, raters have errors and biases that can be revealed. Errors and biases of raters, in many embodiments, are compensated for in order to determine a true intrinsic quality of items. In numerous embodiments, qualities of items and errors and biases of raters are calculated using computer systems. In a multitude of embodiments, computer systems perform iterative computations to solve an estimate of each item's quality and each rater's accuracy and bias until a convergence is reached. In some embodiments, computations that are performed result in a determined true intrinsic quality of each item and an accuracy and bias of each reviewer.

[0089] In various embodiments, a quality of each item is estimated. In several embodiments, when rater error and bias is known, a quality of each item is estimated by Bayesian updating, minimizing some moment function, minimizing the squared errors, or a combination of thereof. In certain embodiments, error σ| and bias bj are utilized to estimate a true quality of an item using the equation: Solving this equation, in accordance with many embodiments, results in an estimate of a quality of an item i:

In particular embodiments, an estimate of the unobserved quality of each item is a sum of the relative ratings given by the experts who rated i weighted by each expert's relative accuracy.

[0090] In many applications, raters' accuracies ¹ ₂ and their biases bj are unobserved.

Accordingly, numerous embodiments are directed to solving an item's true quality when the accuracy and bias of raters are unknown. In some embodiments, multiple ratings of each expert on multiple items are utilized to simultaneously estimate the bias and accuracy of each expert as well as the true qualities of the items.

[0091] In several embodiments, an error (inverse accuracy) of a rater is estimated. In numerous embodiments, an error of a rater is estimated using the equation:

In a number embodiments, a bias of a rater can be estimated. In numerous embodiments, a bias of a rater is estimated using the equation:

[0092] Note that (4)-(6) form a system of n + 2m equations in the same number of unknowns. This system, however, is still under-identified. Accordingly, in several embodiments, the scale on which items' true qualities lie is normalized. In particular embodiments, items' qualities are normalized to to have an average of

In many embodiments, q is chosen by a user. In numerous embodiments, q is selected arbitrarily, as it merely provides an average level from which to interpret qualities and has no effect on estimated accuracies.

[0093] Provided in Fig. 3 is a conceptual illustration of a process to determine a true intrinsic quality of each item and an accuracy and bias of each reviewer utilizing computer systems in accordance with an embodiment of the invention. Accordingly, in the provided embodiment, iterative computations of each item's quality and each rater's accuracy and bias are solved until a convergence is readied, resulting in a determined true intrinsic quality of each item and an accuracy and bias of each reviewer.

[0094] In a number of embodiments, multiple raters' normalized scaled ratings of a number of items to be analyzed are obtained (301). An item, in accordance with several embodiments, is any item rated by a group of raters. In many embodiments, items include (but are not limited to) consumer products, professional services, restaurants, entertainment programs (e.g., movies, theater, music, books), and investment securities.

[0095] The ratable items to be assessed, in accordance with some embodiments, are defined within a category. In various embodiments, a category is defined by the item to be assessed (e.g., movies). In many embodiments, a category is defined by the raters (e.g., Yelp users). It should be noted, however, some processes do not necessitate a categorical definition of a compilation or ratings. In several embodiments, categorical definitions are beneficial to associate a group of items and or raters for comparison.

[0096] In many embodiments, obtained ratings have been normalized such that each rating is on a commensurable scale when compared to the collection of rankings. Any method to normalize the ratings may be used, however, each rating within the collection of ratings should have the same scale, enabling downstream statistical comparison between the ratings. In further embodiments, a collection of obtained ratings is to include at least some overlap between raters and items to be analyzed. Accordingly, various embodiments require that at least two raters of the group of raters each rate at least two items; and that at least two items of the group of items are each rated by at least two raters. In a several embodiments, increased numbers of raters that each rate overlapping groups of items yield a better intrinsic true quality of each rated item.

[0097] Several embodiments are directed to determining an intrinsic true quality of each item, and an error and bias of each reviewer by solving estimations of true quality, error, and bias of each reviewer (See process 300). Because the equations (5) - (7), define a corresponding moment condition that holds with equality at the true parameters ((¾) _» , (bj)j, {<yf)j), the system of equations can be solved utilizing Generalized Method of Moments, in accordance with many embodiments.

[0098] Given that each equation is continuous and nonzero in a neighborhood of the true parameters, standard results show that the solution provides consistent estimators of the true qualities, biases, and accuracies. To ensure compactness, in accordance with various embodiments, qualities, biases, and errors, are restricted to lie in some compact interval of the reals. [0099] For errors, a lower bound that is positive is imposed, in accordance with numerous embodiments, ruling out infinite variance on the part of any expert. In many embodiments, the finite upper bound on accuracy rules out any expert having a null variance (infinite accuracy) and thus always having a rating that is exactly equal to quality. Accordingly, this requires that t{n)— » oo, such that rij > t(n) and mi > t{n) grow for all i, j, so that the number of observed ratings for each item grows (so that item qualities are estimable), and the number of items rated by each expert grows (so that experts' errors are estimable). There is no requirement on the relative size of TO to n, however, various embodiments do require that various ratings grow fast enough. There are some applications in which there are many more raters than items (e.g., online restaurant reviews), and others in which there are more items than raters (e.g., wines rated by experts), all of which can be examined in accordance with a number of embodiments of the invention.

[0100] There are many ways to estimate solutions under GMM, such as parameter grid searches and Markov chain Monte Carlo (MCMC) techniques, which can be used in accordance with a number of embodiments. In some embodiments, a direct technique is utilized, leading to equality by iterating on the system of equations. Accordingly, in a number of embodiments, initial estimates of each rater's error σ| and bias &!· and each item's qualities are determined (303). In a number of embodiments, each rater's error σ| is initiated at some arbitrary positive levels (e.g., all equal to 1). In various embodiments, a rater's bias bj* is initiated by the equation:

And in many embodiments, an item's quality is initiated by the equations:

The last equation rescales to normalize the estimated qualities.

[0101] In several embodiments, estimates of qualities are centered (305) at an overall mean quality, as best currently estimated. In particular embodiments, q* is set to be the best current estimate of the overall average true quality through stage t,

Note that this particular normalization is very useful since it sets the overall average (item and accuracy weighted) estimated bias of the experts to be 0, and thus allows one to interpret bias, according to a number of embodiments.

[0102] In many embodiments, estimates of each rater's error and bias and of each item's quality are solved (307). In particular embodiments, each rater's error and bias and of each item's quality are solved using the equations:

In several embodiments, iterative computations at round t + 1 as a function of the estimates from round t are solved, each iteration centering estimates of items' qualities to the best current estimate of overall average true quality. In a number of embodiments, an an intrinsic true quality of each item, an accuracy and bias of each reviewer is determined (209) by iteratively centering estimates of qualities and resolving estimates of quality, error, and bias, and until convergence. In numerous embodiments, an optional estimation of precision of each item's estimated quality is determined (311). In particular embodiments, an associated estimate of the variance of quality estimate of item is:

This provides a level of confidence in the estimated true value of the item. [0103] In accordance with more embodiments, a converged true intrinsic quality of each item and an error and bias of each reviewer are stored and/or reported (313). Furthermore, in a number of embodiments, normalized true intrinsic qualities may be used in many further downstream applications, including (but not limited to) monetary valuation and item marketing. In several embodiments, the accuracy and bias of each rater is used to evaluate each respective rater. Accordingly, numerous embodiments are also directed to the use of determined rater error and bias to incentivize raters to provide reliable ratings.

[0104] While specific examples of processes for determining the intrinsic qualities of items are described above, one of ordinary skill in the art can appreciate that various steps of the process can be performed in different orders and that certain steps may be optional according to some embodiments of the invention. As such, it should be clear that the various steps of the process could be used as appropriate to the requirements of specific applications. Furthermore, any of a variety of processes for determining the intrinsic qualities of items appropriate to the requirements of a given application can be utilized in accordance with various embodiments of the invention.

[0105] Several embodiments are also directed to a recommender system capable of recommending items based on a user's calculated bias. Accordingly, in many embodiments, a user could generate ratings of various items within a category. A recommender system, in accordance with numerous embodiments, could determine a particular user's bias, utilizing various embodiments described within. Based on a user's bias, according to several embodiments, a recommender system would recommend items that a user may prefer. For example, in the wine industry, a user may have an unrealized bias for wines having extraordinarily dry qualities (e.g., wines with high tannin content). Based on the user's reviews, a recommender system would be able to determine the user's bias for dry wines and make recommendations of wines with high tannin content.

Systems of Intrinsic Quality Valuations

[0106] Turning now to Fig. 4, computer systems (401) may be implemented on computing devices in accordance with some embodiments of the invention. Computer systems (401) may include personal computers, laptop computers, other computing devices, or any combination of devices and computers with sufficient processing power for the processes described herein. Computer systems (401) include a processor (403), which may refer to one or more devices within the computing devices that can be configured to perform computations via machine readable instructions stored within a memory (407) of the computer systems (401). The processor may include one or more microprocessors (CPUs), one or more graphics processing units (GPUs), and/or one or more digital signal processors (DSPs). According to other embodiments of the invention, the computer system may be implemented on multiple computers.

[0107] In a number of embodiments of the invention, the memory (407) may contain an application for acquisition and processing of ratings (409) and an application for determination of true intrinsic qualities of items (411) that performs all or a portion of various methods according to different embodiments of the invention described throughout the present application. As an example, processor (403) may perform a ratings processing method and a quality determination method methods similar to any of the processes described above with reference to Figs. 1 and 2, diiring which memory (407) may be used to store various intermediate processing data such as raw imported ratings (409a), normalized ratings (409b), estimations of quality of items (411a), estimations of error of raters (411b), estimations of bias of raters (411c), and converged solutions (411d).

[0108] In some embodiments of the invention, computer systems (401) may include an input/output interface (405) that can be utilized to communicate with a variety of devices, including but not limited to other computing systems, a projector, and/or other display devices. As can be readily appreciated, a variety of software architectures can be utilized to implement a computer system as appropriate to the requirements of specific applications in accordance with various embodiments of the invention.

[0109] Although computer systems and processes for variant analyses and performing actions based thereon are described above with respect to Fig. 4, any of a variety of devices and processes for data associated with variant analyses as appropriate to the requirements of a specific application can be utilized in accordance with many embodiments of the invention.

[0110] Although the present invention has been described in certain specific aspects, many additional modifications and variations would be apparent to those skilled in the art. It is therefore to be understood that the present invention may be practiced otherwise than specifically described. Thus, embodiments of the present invention should be considered in all respects as illustrative and not restrictive.

EXEMPLARY EMBODIMENTS

[0111] A number of examples are provided to support the methods and systems of determining an intrinsic quality. In the ensuing section, exemplary calculations and applications related to intrinsic quality determination are provided.

Example 1: Expert Ratings of Wines

[0112] Fine wines, and Bordeaux wines in particular, have attracted much interest from economists who aim to identify wine quality and its determinants. Wine is a typical product for which quality differences are simultaneously presumably very large (e.g., prices vary significantly) and difficult to appreciate (as particular wine prices vary significantly from year to year, and even within year for different wines released by the same producer, and there are many producers). Official rankings and expertise have historically played a very important role in the development of these markets. However, experts' opinions have been shown to diverge even within relatively homogeneous sub-segments of the market.

[0113] Key parts of the Bordeaux fine wine industry operate via a futures/forwards market. At specific points in the season, wines that are not yet even bottled are tasted and rated by trained professionals and experts. Their ratings are vital for intermediaries and investors who will buy most of the production. Many of these ratings are eventually published in various media (magazines, books, websites). The wine will only be bottled and transferred to the buyers one to several years later (depending on the aging policy of the producer). The empirical study in this example focuses on such ratings of "en primeur" wines because these ratings are less likely to be polluted by cross influences and other information, as they are the first ratings and are essentially simultaneous.

[0114] A database of 52,968 "en primeur" ratings from 19 experts was used for this study. They are wine critics, journalists, writers, and bloggers. Some like Robert Parker and Jancis Robinson are world-renowned critiques. In some cases an expert has issued multiple ratings of the same wine and vintage. In those cases, a mean rating was used. This results in 51,862 ratings. When the experts' ratings were rescaled to lie in the same interval (see below), the bottom five percent of the wines were dropped, which are often in a long bottom tail of wines that are often somehow defective, sometimes idiosyncratically. This eliminates 1,825 wine/vintages. The analysis is robust to this dropping. Another 5,687 wine/vintages were dropped that are rated by only one expert. In sum, 44, 350 ratings of n = 6, 243 wine/vintages (with vintages from 1998 to 2015) given by the m = 19 experts were used in this analysis (Fig. 5).

Normalizing the Scales of Wines

[0115] Different wine experts use different scales for their ratings. For instance, Parker rates wines from 50 to 100, but essentially only ever rates between 70 and 100. Jancis Robinson employs a scale from 1 to 20 and usually rates between 10 to 20.

[0116] To adjust for these different scales, all experts ratings were converted to lie on a 0 to 100 scale and to use the whole scale. First,the lowest five percent of each expert's ratings were dropped, as the lower tail is quite long and noisy. Each expert's ratings was then linearly rescaled so that their lowest rated wine is given a rating of 0 and the highest rated wine is given a rating of 100. [0117] Letting G denote the raw scores of the experts, the rescaled ratings are:

[0118] Given that some experts use a coarser scale than others, there are obvious peaks in their distribution. For instance, if they use a 20 point scale with half points rather than 100 point scale, then 19.5 becomes 97.5, 19 becomes 95, etc., and so there are clumps at certain points on the 100 point scale that are used (See Figs. 6 and 7).

Convergence and Intrinsic Qualities of Wine

[0119] Convergence of the algorithm to the GMM solution was very rapid. Generally, the solution was reached after 5 to 10 iterations and full equality of the equations was hit and then there was no further movement of the remaining of the 100 iterations (See Figs. 8 and

9).

[0120] The sensitivity of the estimations to a number of parts of the process was examined. For instance, the form of initial rescaling of the ratings or setting the initial experts qualities to unity have no significant impact on the results. Using the estimated sigma squared provided identical results.

[0121] Provided in Fig. 10A are the biases of the experts.

[0122] As accuracy = 1/σ| is hard to interpret directly, the formula was normalized by multiplying the average variance of the experts σ|,/TO. Thus, an expert with an average accuracy will show up as having accuracy 1. An expert with accuracy 2 has twice the average precision, and so forth (Fig. 10B).

[0123] The correlation of an expert's ratings are with the estimated true quality of the wines s/he rates can also be measured. The correlation of an expert's prediction of the quality of a wine is related to the expert's accuracy.

[0124] be the variance in the quality of a typical wine. Note that

Therefore,

[0125] Thus, since accuracy and correlation is , the two are very similar

functions.

[0126] Note that this correlation is not estimable without using this method, since one needs to estimate the quality of the wines to estimate the correlation of an expert's ratings with that quality.

[0127] One can see this close relationship between accuracy and correlation in Fig. 11.

[0128] Recall that this model presumes that the experts' accuracies are independent of the quality of a wine - so they are just as good or bad at rating a high quality wine as a low quality wine. In essence it is assumed that One might expect that experts'

errors would increase when wines are of lower quality; or one might even expect the opposite. The relation between the estimated wine qualities and errors (Table 1, Fig.12). One can see little relationship between errors and quality from the tenth to ninety-fifth percentile of item quality, that is for most middle-quality wines. There is a slight decrease of the average error for the highest and lowest five percent of rated wines.

[0129] The top-100 wines from the sample along with their estimated qualities is provided in Table 2. The number one Bordeaux wine is actually a Sauterne (sweet white wine), Chateau Yquem 2009, and Chateau Marguaux 2010 is the best red wine.

[0130] As the determined qualities use the full 100 point scale and have an average in the 30's, the reported qualities may "look" unfair as most of the consumers and experts have the most known experts' ratings distribution in mind. For instance, most people have an idea of what an 80 or 90 point rating of a wine means according to Robert Parker. For instance, it would probably sounds weird to any professional in the fine wine industry to give a less than 90 point rating to a Lafite Rotschild 2010. To avoid potential misunderstanding due to interpreting wine qualities in the scales that people are often used to, the quality ratings are also rescaled to place them back in the subregion of the 100 point scale usually used by wine experts - who rate almost all wines between 70 and 100. To do this, a "Parker-equivalent" quality level was calculated, which uses the same part of the scale that Parker ususally uses.

[0131] Figure 13 shows how the distribution of ratings on the 100 points scale is modified when rescaled to a "Parker nominal view". Note that this of course does not modify at all the ranking of the wines - it is just a shifting and renormalizing of the scale. This modified quality is reported in the second column (entitled "rescaled") of Table 2.

[0132] As Bordeaux wineries are best-known for their red wines, a separate ranking restricted to that subsample is also provided. The results are presented in Tables 3 and 4.

rank 4i Rescaled wine vintage Type appelation claesement

71 77,77 98 Palmer 2015 Red Margaux Troisieme Cru Classe en 1855

72 77,77 98 Vieux Chateau Certan 2015 Red Pomerol Grands Pomerol

73 77,44 98 Eglise Clinet 2010 Red Pomerol Grands Pomerol

74 77,43 98 Margaux 2003 Red Margaux Premier Cru Claese en 1855

75 77,22 98 Graiid Vin de Latour 2004 Red Pauillac Premier Cru Classe en 1855

76 77,14 98 Doisy Daene, l'Extravagant 2015 Sweet Sauternes Deuxleme Cru Cli isse en 1855

77 76,86 98 Leovllle Las Canes 2005 Red Saint Julien Deuxieine Cru Classe en 1855

78 76,60 98 Doisy Daene, l'Extravagant 2006 Sweet Sauternes Deuxleme Cru Cli isse en 1855

79 76,52 98 Pontet Canet 2009 Red Pauillac Cinquieme Cru Claese en 1855

80 76,33 98 Angelus 2015 Red Saint Einilion Grand Cru Premier Cru Claese A

81 76,22 98 Ausone 2008 Red Saint Emilion Grand Cru Premier Cru Claw le A

82 75,82 98 Vieux Chateau Certan 2009 Red Pomerol Grands Pomerol

83 75,74 98 Trotanoy 2009 Red Pomerol Grands Pomerol

84 75,73 98 La Mission Haut Brion 2009 Red Pessac Leognan Grand Cru Classe de Graves (Rouge)

85 75,72 98 Doisy Daene, 1' Extravagant 2005 Sweet Sauternes Deuxleme Cru Cli isse en 1855

86 75,63 98 Leovllle Las Cases 2010 Red Saint Julien Deuxleme Cru Cli ksse en 1855

87 75,57 97,5 Petrus 2012 Red Pomerol Grands Pomerol

88 75,56 97,5 Yquem 2010 Sweet Sauternes Premier Cru Clasi ■e en 1855 - Sauternes

89 75,44 97,5 Yquem 2006 Sweet Sauternes Premier Cru Claw le en 1855 - Sauternes

90 75,26 97,5 Palmer 2010 Red Margaux Troisieme Cru CI: isse en 1855

91 75,18 97,5 Cos d'Estournel 2010 Red Saint Estephe Deuxleme Cru Cli isse en 1855

92 75,11 97,5 Eglise Clinet 2015 Red Pomerol Grands Pomerol

93 75,10 97,5 Pavie 2005 Red Saint Emilion Grand Cru Premier Cru Claw le A

94 75,06 97,5 Grand Vin de Latour 2014 Red Pauillac Premier Cru Clasi ■e en 1855

95 74,85 97,5 Yquem 2004 Sweet Sauternes Premier Cru Claw le en 1855 - Sauternes

96 74,83 97,5 Montrose 2009 Red Saint Estephe Deuxleme Cru Cli ksse en 1855

97 74,78 97,5 Haut Bailly 2015 Red Pessac Leognan Grand Cru Classe de Graves (Rouge)

98 74.64 97.5 Doisy Daene, l'Extravagant 2003 Sweet Sauternes Deuxleme Cru Cli isse en 1855

99 74,60 97,5 Pavie 2015 lied Saint Emilion Grand Cru Premier Cru Classe A

100 74.46 97.5 Climens 2005 Sweet Sauternes Premier Cru Claw le en 1855 - Sauternes

Table 3 Ranking experts for red Bordeaux only.

expert normalized accuracy = = (<¾,,, %) rij

»ϊ — O- Corr

Antonio Galloni 145.1364 .8583284 .7917423 954

Bettane & Desseauve 144.4908 .8621639 .8083333 2520

Chris Kissack 145.3225 .8572295 .7804236 1886

Decanter 78.21471 1.592728 .8924085 1879

JM Quarin 52.87291 2.356116 .90062 2402

Jacques Dupont 194.0192 .6420742 .7331077 2492

Jacques Perrin 78.1182 1.594695 .9052212 419

James Suckling 132.6135 .9393818 .8185688 1650

Jancis Robinson 184.6749 .6745623 .6930351 2965

Jeannie Cho Lee 127.0811 .9802777 .833195 1001

Jeff Leve 91.018 1.368682 .8921425 1336

La RVF 155.5768 .8007281 .8070636 1724

Neal Martin 141.0504 .8831929 .808379 2371

Rene Gabriel 158.4639 .7861395 .7969397 3972

Robert Parker 102.5979 1.214203 .8505908 2461

Tim Atkin 199.3397 .6249368 .7382556 1506

Wine Enthusiast 179.049 .6957576 .7653303 2003

Wine Spectator 187.6129 .6639986 .8094715 2961 Yves Beck 205.9752 .6048046 .7796345 378

Example 2: Establishing Price Based on Intrinsic Quality

[0133] Generally, there is a textbook identification problem that stems from the fact that prices are determined by both supply and demand, which can both move to affect prices. Here, identification comes from the fact that prices are largely determined after the amount of wine supplied is already largely fixed, and then the quality of the wine is later made known and prices result. Thus, supply is treated as inelastic, and prices reflecting perceived quality. Moreover, by including various fixed effects, it is deviations in prices that are being attributed to relative qualities of the wines.

[0134] Prices cannot simply be regressed on the estimated quality because other factors influence the posted prices. For instance, shops' attributes, vintages, local production origins (AOC) and official rankings are clearly observed by the consumers and are likely to affect the prices, holding wine quality constant. Even if these variables are also correlated with unobserved quality, it is possible to control for them to obtain a lower bound of the correlation between the calculated quality and prices.

[0135] Consumers may also observe and be directly influenced by some experts' ratings. In the wine industry, it has been shown that Parker ratings have a direct and significant impact on prices. Omitting such variables could lead prices to correlate with the estimated quality simply because the quality estimates are also positively correlated with expert ratings that consumers and wine shops observe. The problem reverses a traditional question addressed in the wine economics literature which aims to identify the causal impact of the ratings on the prices when wine quality is unobserved. Instead, the relationship between our calculated wine quality and prices is estimated, and then controlled for salient information.

[0136] In the Bordeaux wine industry, quantities are completely fixed for a given vintage (production cannot be significantly adjusted upward by mixing the wine of that vintage with wine from other vintages). The main adjustment to an increased individualized demand is thus on the price. Thus, an hedonic (price) regression is estimated.

[0137] It has been shown that wine prices are affected by the weather conditions at crucial points in the season in the production year and by wine aging. Such weather conditions were controlled for by including vintage-appellation fixed effects: dummies that capture the weather conditions for various vintages in the specific sub-region of Bordeaux production. The sale year and retail shop fixed effects were also included, which can influence the observed prices. An Official ranking' fixed effects was included, (see Table 5).

Table 5: Official rankings

[0138] Ratings of well-known experts are likely a direct impact on prices. Accordingly, in this exemplary method the salient "reference" experts' ratings are controlled by directly including the ratings of the best-known expert for Bordeaux fine wines, Robert Parker. The ratings of Jancis Robinson, who is another big name for Bordeaux wines, were also included.

[0139] In some regressions, the "best" rating of each wine was also controlled for, as in retail stores, sellers often transmit to the consumers the most favorable piece of information so as to influence their decisions.

[0140] Lastly, as a limit experiment, the average rating among experts (properly nor- malized) was used as a supplementary control to check whether the estimated quality still significantly explains price variation when controlling for the average rating.

[0141] All the ratings used are corrected to span the 1-100 scale as exposed in Equation

16).

Pricing of Wine

[0142] The Bordeaux wine "terroir" is typically documented by sub appellations such as Medoc, Saint Emilion, Premieres Cotes de Bordeaux or Pauillac. These appellations are very much linked to the notion of terroir as they relate to specific sub-regions of production as well as (most of the time) typical production constraints (types of grapes, specific production quantifies per hectares...). The Bordeaux wine is also associated to official ranking such as Crand Cm Classe 1855 or Premier Grand Cru (see Table 5).

[0143] The prices of the wines are from surveys of restaurants in three of the main worldwide markets: in Hong Kong, New York and Paris (Table 6, Fig. 14). The prices were recorded between 2010 and 2016. Initially, 93,466 prices of standard bottle Bordeaux wines were recorded.

[0144] Each wine/vintage rated en primeur was matched with all posterior prices and obtained a database of wine/vintage prices observations, in a given shop and year. Out of the 2,439 wine/ vintage that were considered, there were 39,678 such observations, that is 16.27 prices on average for each wine/vintage (Table 7).

Table 6: Markets surveyed, stores and prices

Table 7: Wines by Appellation

[0145] Figure 14 shows the price distributions in the three markets. Table 8 lists the top- 100 most surveyed restaurants in the data. Table 8: Top 100 most surveyed stores (restaurants)

Store Market Number of Wines Number of Prices

L'Atelter eta Joel Rohuchon - HK Hong Kong 429 1607

La Truffiere Paris 409 1270

Lo Cinq - Paris Paris 288 581

Le Carre doe Feuillante Paris 272 1077

Apicius Paris 272 397

Le Pre Catelan Paris 263 431

Petrus - HK Hong Kong 237 917

Epicure Paris 234 370

Cepage Hong Kong 223 507

L Abeille (Shangri-La) Paris 190 558

Per Se New York 172 265

KO Dining Group (Messina, Yu Lei, Kazuo Okuda) Hong Kong 171 608

Mandarin Oriental Paris - Sur Mesure, Camelia Paris 159 411

Le Meurico Paris 156 410

21 Club New York 154 394

Shang Palace (Shangri-La) - Paris Paris 154 282

Au Trou Gascon Paris 147 505

The Steak House winebar + grill Hong Kong 137 321

Alain Ducasse au Plaza Athenee Paris 136 326

Spoon Hong Kong 136 281

Le relais du plaza (plaza athenee) Paris 132 149

Le Grand Vefour Paris 131 276

Yan Toh Heeu Hong Kong 129 241

The Modern New York 129 180

Aureole New York 128 246

Amber Hong Kong 125 191

Bit Steak New York 124 171

Le Diane Paris 118 232

Pierre - HK Hong Kong 116 288

Pouquet's Paris 115 180

Sparks Steak House New York 115 407

Mandarin Bar and Grill Hong Kong 115 246

Tin Lung Heen Hong Kong 113 182

Daniel New York 112 277

Man Wah Hong Kong 107 221

Eleven Madison Park New York 107 174

Morrell Wine Bar & Cafe New York 104 144

City Winery New York 103 151

Shang Palace - HK Hong Kong 102 356

Porter House New York 101 147

Jean Georges New York 101 136

Veritas New York 98 253

Asiate New York 96 191

Jean- Francois Piege Paris 95 95

Le Cirque New York 91 129

Mathieu Pacaud - Histoires Paris 91 91

Pierre Gagnaire Paris 91 129

Conrad Hotel (Golden Leaf) Hong Kong 91 152

Hexagone Paris 90 90

Caprice Hong Kong 89 284

The Mark Restaurant by Jean-Georges New York 88 134

Benoit - Paris Paris 88 114

Cafe Boulud New York New York 83 131

G Bar Hong Kong 83 178

Le Gabriel - Paris Paris 81 81

Harlan's Hong Kong 81 160

Le Bernard in New York 81 11 1

Gordon Ramsay Au Trianon Paris 81 81

Sewa Hong Kong 81 81

Pur' Paris 80 121

Guy Savoy Paris 79 79 Store Market Number of Wines Number of Prices

Chez Flottes Paris 79 153

Tosca - HK Hong Kong 79 143

L'Altro - HK Hong Kong 79 168

Bouley New York 78 10S

Plchol!ne New York 77 99

A Voce - Columbus New York 77 181

Hotel Park Hyatt- Paris Vendome Paris 76 101

Angelinl Hong Kong 76 76

Nice Matin New York 76 181

Lili au Peninsula Paris 73 116

Ming Court Hong Kong 73 218

La Compagine des Vine surnaturels Paris 73 83

Fook Lam Moon - Hong Kong Hong Kong 73 141

Drouant Paris 72 90

Bibo Hong Kong 71 71

Bit Prime - NYC New York 71 121

La Table du Lancaster Paris 70 127

Le Violon d'Ingres Paris 70 91

NOBU Intercontinental Hong Kong Hong Kong 70 171

Gabriel Kreuther New York 69 69

Michel Rostang Paris 68 68

L'Atelier de Joel Robuchon - Paris Paris 68 99

Cuisine Cuisine at Mira Hong Kong 68 135

Smith & Wollensky New York New York 68 96

Mandarin Oriental (Krug Room) Hong Kong 68 101

Cuisine Cuisine at IFC Hong Kong 68 75

Le Beef Club / Pish Club Paris 67 84

La Grande Cascade Paris 67 67

Nicholini'a Hong Kong 67 70

Dominique Bouchet Paris 66 93

Gotham Bar and Grill New York 64 100

Benoit - New York New York 64 86

Lung King Heen Hong Kong 64 163

Lee 110 de Tai lie vent Paris 63 118

Rouge Tomate New York 63 120

Harrys Cafe and Steak New York 63 128

Le Celadon Paris 63 108

La Scene - Hotel Prince de Galles Paris 62

62

[0146] As prices axe likely to be correlated across observations of the same wine, all errors are clustered at the wine/vintage level.

[0147] The results (see Table 9) show that the estimated quality is a significant predictor of prices. Its coefficient is positive and always significant at the 1 percent level (in four estimations, including our preferred one, it is significant at the .001 level); noting that a number of fixed effects have been included such as vintage x appellation, official ranking, price year and store fixed effects; and even given other controls.

[0148] In the regression of column 5, the estimated quality is the only one which significantly explains the price. Interestingly, Robert Parker who is often considered as a "guru" ends up having no significant influence on prices after controlling for estimated quality. Only Jancis Robinson's ratings are positive and significant at the 5 percent level.

[0149] As prices and ratings are in logs, the coefficients can be interpreted as elasticities. In the regression of column 5, the elasticity of the estimated quality on prices is very high: a 10 percent increase in quality raises the price of 14 percent - so there is an elasticity of 1.4 of price on our quality rating.

[0150] Even when the average rating is introduced on the top of the best rating and the other salient experts' ratings (column 6), the significance of the quality estimate remains significant while the average rating is not. Thus, our rating provides significant predictions and ones that are not captured in the average rating. Note that this is even though the average rating is already incorporating an adjustment in which all experts' ratings are put on the same scale. So, it is the adjustments for accuracy and bias that are what are providing the predictive power.

Table 9: Retail prices as a function of estimated wine quality and salient and best en primeur ratings.

(1) (2) (3) (4) (5) (6)

Estimated quality 0.736+ 0.451* 0.676+ 1.235+ 1.426+ 1.028*

(8.39) (3.08) (4.13) (16.66) (9.30) (2.60)

Best rating 0.500+ -0.0602 -0.111

(3.56) (-0.43) (-0.76)

R. Parker rating 0.207# 0.0286 0.0514

(2.13) (0.45) (0.77)

J. Robinson rating 0.0953+ 0.0689* 0.0587

(3.43) (2.16) (1.77)

Average rating 0.427

(1.13)

N 39678 39666 34069 24468 21145 21145 r2 0.785 0.790 0.777 0.824 0.828 0.828 aic 54779.4 53850.3 47143.5 27585.1 23372.5 23355.2 bic 56926.5 55997.4 48948.8 29084.5 24709.6 24700.3

Notes: l!-statistics are in parentheses. The standard errors are clustered at the wine x vintage level. Significance levels: # p < 0.05,* p < 0.01,+ p < 0.001. All variables (dependent and explaining) are in logs so that coefficients can be interpreted as elasticities. All regressions include vintage x appellation, official ranking, price, year, and store fixed effects. Ratings are corrected to span the 1-100 scale (see Equation 16).

[0151] Estimated wine qualities are correlated with retail prices, controlling for many things (including ratings). This is reassuring as it tends to confirm that prices do reflect item quality as captured by this exemplary technique. To what extent do individual expert's ratings correlate with prices as a function of how accurate they are.

[0152] It is expected that more accurate experts are to have a greater correlation of their ratings with the prices, since their ratings correlate more strongly with the estimated true quality which correlates with prices. However, there are many other factors which may affect the correlation of prices with the ratings. To control for that log prices on each expert's logs ratings were separately regressed (see raw results in the Table 10). It is noted that several experts, such as Beck, Galloni, Lee, Leve, Perrin and Suckling, could not be considered as too few of their ratings were for wines with observed prices. Among the thirteen remaining experts, the most accurate expert, JM Quarin is also the one whose ratings correlate most with the prices: a 10 percent increase in his ratings corresponds to a 7.1 percent increase in prices. Parker, who is the second most accurate in this list has the second highest correlation between ratings and prices (a 10 percent increase in his ratings corresponds to a 6.8 percent increase in prices).

Table 10: Retail prices as a function of "en primeur" ratings by the top-5 most influential experts (on prices). All markets.

Notes: i-statistics are in parentheses. The standard errors are clustered at the winexvintage level. Significance levels: # p < 0.05,* p < 0.01,+ p < 0.001. All regressions include vintage, re-rating year, vintage x appellation and official ranking. Ratings are corrected to span the 1-100 scale (see Equation 16). Prices and ratings are in log so that coefficients can be interpreted as elasticities. Only the experts who have rated wines for which we have a sufficient number of prices (>2000) are considered here.

[0153] Figure 15 shows that the correlation between an expert's ratings and prices increases with the expert's accuracy. In addition, some experts lie above or below the line. They have a residual correlation with price that goes beyond what is predicted by their accuracy (which correlates with prices because of the the strength of their ratings' correlation with quality). This residual correlation could reflect different things. Here are two possibilities. It could be that the expert's rating influences the price, as is often claimed, for instance, about Parker's ratings. It could also be that the expert's rating is affected by the anticipated price point that a wine will sell at - giving higher ratings to more expensive wines (after adjusting for quality).

Table 11: Retail prices as a function of "en primeur" ratings by the top-5 most influential experts (on prices). Paris market.

(1) (2) (3) (4)

JM Quarin 0.588+ 0.543+ 0.486+ 0.471+

(8.64) (7.97) (5.69) (5.49)

Robert Parker 0.282 ¹ 0.211* 0.400+ 0.412+

(4.11) (3.16) (3.84) (3.72)

Rene Gabriel 0.313+ 0.240* 0.218*

(6.14) (3.28) (2.79)

Wine Enthusiast 0.384+ 0.391+

(6.23) (6.20)

Bettane & Desseauve 0.153 ^#

(2.33)

N 7438 7189 4248 4040

r2 0.808 0.813 0.859 0.863 aic 8439.4 8012.2 4253.5 3971.1 bic 9296.8 8851.6 4895.3 4595.2

Notes: i-statistics are in parentheses. The standard errors are clustered at the wine x vintage level. Significance levels: # p < 0.05,* p < 0.01,+ p < 0.001. All regressions include vintage, re-rating year, vintage x appellation and official ranking. Ratings are corrected to span the 1-100 scale (see Equation 16). Prices and ratings are in log so that coefficients can be interpreted as elasticities. Only the experts who have rated wines for which we have a sufficient number of prices (>2000) are considered here. Table 12: Retail prices as a function of "en primeur" ratings by the top-5 most influential experts (on prices). New York market.

(1) (2) (3) (4)

JM Quarin 0.479+ 0.406+ 0.531+ 0.498+

(7.59) (6.53) (6.27) (5.97)

Robert Parker 0.664+ 0.560+ 0.492+ 0.460+

(7.45) (5.86) (4.40) (3.95)

Rene Gabriel 0.322+ 0.301+ 0.289+

(5.07) (4.03) (3.72)

Wine Enthusiast 0.370+ 0.366+

(6.20) (5.91)

Bettane & Desseauve 0.157

(1.83)

N 4247 4145 2765 2701 r2 0.832 0.839 0.871 0.873 aic 3948.8 3662.5 2113.2 2042.2 bic 4590.6 4301.8 2599.0 2526.1

Notes: ^-statistics are in parentheses. The standard errors are clustered at the wine x vintage level. Significance levels: # p < 0.05,* p < 0.01,+ p < 0.001. All regressions include vintage, re-rating year, vintage x appellation and official ranking. Ratings are corrected to span the 1-100 scale (see Equation 16). Prices and ratings are in log so that coefficients can be interpreted as elasticities. Only the experts who have rated wines for which we have a sufficient number of prices (>2000) are considered here.

Table 13: Retail prices as a function of "en primeur" ratings by the top-5 most influential experts (on prices). Hong Kong market.

(1) (2) (3) (4)

JM Quarin 0.691+ 0.606+ 0.626+ 0.587+

(6.97) (5.72) (4.17) (4.03)

Robert Parker 0.625+ 0.568+ 0.613* 0.555*

(4.70) (4.04) (3.25) (2.74)

Rene Gabriel 0.312+ 0.290* 0.305*

(3.32) (2.67) (2.75)

Wine Enthusiast 0.521+ 0.513+

(6.26) (5.76)

Bettane & Desseauve 0.195

(1.88)

N 4684 4608 2836 2772

r2 0.770 0.774 0.842 0.844 aic 6538.5 6361.5 3092.3 3015.0 bic 7170.7 6992.2 3556.4 3471.5

Notes: ^-statistics are in parentheses. The standard errors are clustered at the wine x vintage level. Significance levels: # p < 0.05,* p < 0.01,+ p < 0.001. All regressions include vintage, re-rating year, vintage x appellation and official ranking. Ratings are corrected to span the 1-100 scale (see Equation 16). Prices and ratings are in log so that coefficients can be interpreted as elasticities. Only the experts who have rated wines for which we have a sufficient number of prices (>2000) are considered here.

Example 3: Experts' Biases and Accuracies that vary with Categories of Items

[0154] Any reviewer's ability and judgment in rating items might vary with categories of items. There is no reason to expect that an expert who is extremely accurate in reviewing wines would be a good analyst for recommending movies or cars or stocks. In one scenario, it might be that an expert on wines is much better at judging red wines than white wines, or judging Bordeaux wines than Spanish wines. The distinctions do not end there: even within Bordeaux there are distinctly different red wines. The wines from the "left bank" (the west side of the Gironde Estuary) and the "right bank" (the east side), generally contain different blends of grapes and come from different soils and can even have different weather conditions. The left bank wines are blends that predominately feature Cabernet Sauvignon grapes, while the right bank wines tend to feature Merlot grapes, with varying mixtures and often including Cabernet Franc and other grapes. While not as different as red from white, there are still sufficient distinctions that make these two categories different from each other and it can be that a given expert would favor Cabernet Sauvignon over Merlot grapes, or vice versa. This might result in different biases and/or accuracies for the two regions.

[0155] Effectively any given expert can be treated as two completely different experts, one for Left Bank Bordeaux and one for Right Bank Bordeaux. One of those two experts might have a large positive bias and the other a slight negative bias, and correspondingly one might be very accurate and the other more variable.

[0156] One could interpret the biases as "preferences": a deviation from the average "true" quality that favors or goes against a certain type of wine.

[0157] Thus, for any given set of items N, one can partition that set, and treat each distinct group as a completely different set of items and run the method separately on that set of items. Thus, for every reviewer, a different bias and accuracy are determined for every category of items.

[0158] To illustrate this, the data on Bordeaux winesLeft Bank and Right Bank wines was split and analyzed.

[0159] Let L denote "Left Bank" and N\L denote "Right Bank". Left vs Right Bank Tastes of Experts

[0160] The estimation was ran for all experts separately on the left and the right bank. Formally, the evaluations of any expert j are:

This leads to the following results (Figs. 16A 16B).

[0161] The differences of estimated biases across the left vs right dichotomy can be computed:

The differences in accuracies can also be computed:

Let

denote the normalized accuracy of expert j on the Left Bank wines, and similarly define the Right Bank accuracies A _jj f\i. The difference in expert j's normalized accuracies between Left Bank and Right Bank is then: [0162] Results are provided in Fig. 17.

[0163] One can see that Robert Parker a "rightist," which is consistent with him being known for advocating in favor of powerful Bordeaux wines, mostly located on the right bank. Other pronounced "rightists" include Jeff Leve, James Suckling, Chris Kissack, Wine Spectator and Yves Beck. On the other side, Decanter, Jacques Dupont, La RVF, Jancis Robinson, Wine Enthusiast, and Bettane & Desseauve favor more traditional and reserved wines. This could explain the lack of correlation between Parker's and Robinson's ratings which is presumed to be due to different preferences in wine "styles".

[0164] It is also interesting to explore how the differences in accuracies relate to the differences in biases. This relationship is portrayed in Fig. 18.

A Significant Difference [0165] Utilizing methods described herein, one can test whether there is a significant difference in Left Bank and Right Bank wines by examining whether there is a significant increase in the predictions of qualities.

[0166] First, the residual weighted sum of squares is defined for the different ways of estimating.

[0167] Without any distinction between Left and Right Bank wines, the overall weighted sum of squared errors from keeping all the wines in one category was:

The adjustment by A weights the terms so that the errors are all nor-

malized to have the average variance and thus the same distribution - which is the same as weighting each estimate by its relative precision which produces the overall estimated sum of squared errors. Since

this becomes

[0168] Once divided into two categories,a second sum of squared errors is calculated:

Using the similar calculations as for Equation 22, it comes:

noting that all experts are rating wines on both Left and Right Banks, and so there is no subscripting on m.

[0169] The results identify n = 36, 821 ratings of red wines into one of the Left or the Right bank (some wines blend grapes from both sides of the river and the origins of some others is not clear in the data). These divide into rii = 19, 560 ratings of Left Bank wines and 17,261 of Right Bank wines. Then, with the data, it is determined that

2,699.726 = 5,231,926, and R

[0170] There are 38 parameters estimated in the original algorithm and 76 parameters estimated in the algorithm in which wines were split into Left and Right Banks. This results in an F-test statistic of:

[0171] At a 1 percent significance level, the F-test threshold with (38; 36, 744) degrees of freedom is 1.59. The F statistic of 17.487 greatly exceeds that threshold value. Thus, there are significant differences in experts' rating patterns for Left and Right Bank wines.

Example 4: Possible Micro-Foundations for the Empirics

[0172] In this example, a couple of simple models that would micro-found the reduced form regressions on prices are presented. As such, these models introduce specific assumptions that are not necessary, but provide one possible rational.

Prices

[0173] A wine has an unobserved quality q that is a function of some fundamentals / and of an independent term φ:

An expert observes the fundamentals and a noisy signal of the other term: with

The expert rates the item as

with denoting the expected quality conditioned on the observed s ^r and /. This

would be a typical "en primeur" rating of a Bordeaux wine, which most of the time isn't blind. Note that the bias is not considered here to keep the notation uncluttered, but introducing it would be straightforward (just add it into the rating above).

[0174] Consumers are unbiased and can also observe the fundamentals. If the consumers aggregate a set of noisy and independent signals that provide information about the term φ, then one can capture their expectation as

[0175] Regardless of how many ratings a consumer observes, because of the salience of some particular expert's rating, the consumer could also directly be influenced by that rating. The consumer may also be influenced by other factors such as the information printed on the bottle, e.g. the brand, the appellation and the official ranking. A simple way of thinking of this problem is to mix these factors, so that with some weight or probability λ the consumers base their expectation on a set of observed reviews S, with weight or probability μ they follow the signal on quality contained in the public information (the brand, appellation or official ranking) a, and with the remaining weight or probability they follow the salient

expert's rating. The conditional expected quality or random consumer is then given by

where q is the best estimate of q given S (e.g., as the one we developed here), and e is an error term.

[0176] In the Bordeaux wine industry, quantities are completely fixed for a given vintage (production cannot be significantly adjusted upward by mixing the wine of that vintage with wine from other vintages). The main adjustment to increased demand is via prices. We therefore estimate an hedonic (price) regression of the form: g where is an increasing function that gives a price to a "perceived" quality in the market. For example:

where is assumed to be linear with slope 0, and with β The other terms of the right hand side of Equation (28) control for effects found in the literature so far. The term i/ _a denotes the official ranking fixed effect. A fundamentals fixed effect is added because it is likely that the fundamentals are not perfectly observed by the expert and could influence the price. The two other fixed effects, and capture the selling

year and the retail store specifics that may also affect the posted price.

[0177] The coefficients β and β ^τ are parameters of interest. It is conjectured that the measure of true quality impacts prices, and so even when controlling for all determinants including for some salient experts ratings, β should remain positive and significant. Some of the previous literature suggests coefficient β ^τ may also be positive and significant.

Re-ratings

[0178] Next, consider a situation in which an expert, who already rated a wine/vintage "en primeur", re-rates that same wine. The expert observes two signals, if in the first period (en primeur), as well as a new conditionally independent signal s', so that 8 = φ+ β. and s' = <f>+e' with e, e' ~ Φ (Ο, σ) and e'_l_e. In the first period, every thing works as before, that is as in Equation 26 (dropping r superscripts). In the second period, the expert's rating is may be dependent upon her own previous signal. Moreover, the expert could be also influenced by peers, and in particular by the most prominent ones. Therefore the expert's re-estimation of quality is which is conditioned on the fundamentals /, the previous signal s, the new signal s', and the "reference expert" rating ^r (which, for instance, leads the expert to know the other prominent expert's signal The new rating g is thus given by:

[0179] Again, as a simplifying assumption, suppose that the expert weights the first signal with prob λ, the new signal with prob μ, and the reference expert signal with prob (1— λ— μ). Equation (29) becomes

Using Equations 25 and 26, this becomes:

Rearranging:

where denotes official ranking fixed effects and i a vintage/appellation fixed effect that captures the fundamentals. The term accounts for the re-rating year and v _e is an expert fixed effect. The error term e' is an error term.