Title:
DYNAMIC EXPERIMENTATION EVALUATION SYSTEM
Document Type and Number:
WIPO Patent Application WO/2019/143543
Kind Code:
A3
Abstract:
A dynamic experimentation evaluation system provides a framework in which a continuous stream of metric data is monitored to establish a causal relationship between changes in a software program and the effect of user-observable behavior. In one aspect, an A/B test is performed continuously on a stream of metric data representing the usage of a control version of software product and the usage of a treatment version of the software product. A sequential probability ratio test (SPRT) is used as the test statistic to determine when to terminate the test and produce results within a specific confidence interval and controlled error rate.
Inventors:
SUNDARESAN NEELAKANTAN (US)
YAO CENZHUO (US)
YAO CENZHUO (US)
Application Number:
PCT/US2019/013420
Publication Date:
October 31, 2019
Filing Date:
January 14, 2019
Export Citation:
Assignee:
MICROSOFT TECHNOLOGY LICENSING LLC (US)
International Classes:
G06F11/36; G06F11/34
Foreign References:
| US20140278198A1 | 2014-09-18 | |||
| US20150012852A1 | 2015-01-08 | |||
| US7171586B1 | 2007-01-30 | |||
| US20170323331A1 | 2017-11-09 | |||
| US9141522B1 | 2015-09-22 |
Other References:
ANONYMOUS: "Sequential probability ratio test - Wikipedia", 10 February 2017 (2017-02-10), XP055586086, Retrieved from the Internet [retrieved on 20190507]
RAMESH JOHARI ET AL: "Peeking at A/B Tests", 13 August 2017, KNOWLEDGE DISCOVERY AND DATA MINING, ACM, 2 PENN PLAZA, SUITE 701NEW YORKNY10121-0701USA, PAGE(S) 1517 - 1525, ISBN: 978-1-4503-4887-4, XP058370722
RAMESH JOHARI ET AL: "Peeking at A/B Tests", 13 August 2017, KNOWLEDGE DISCOVERY AND DATA MINING, ACM, 2 PENN PLAZA, SUITE 701NEW YORKNY10121-0701USA, PAGE(S) 1517 - 1525, ISBN: 978-1-4503-4887-4, XP058370722
Attorney, Agent or Firm:
MINHAS, Sandip S. et al. (US)
Download PDF: