Login| Sign Up| Help| Contact|

Patent Searching and Data


Title:
SYSTEM AND METHOD FOR ESTIMATING THE QUALITY OF NETWORKS OR SERVICES THROUGH CROWDSOURCING
Document Type and Number:
WIPO Patent Application WO/2021/009134
Kind Code:
A1
Abstract:
The invention provides a system and method offered by a measurement system provider, for measuring the quality or other characteristics of one or more networks and/or one or more services that are transmitted over said networks by means of measurements performed by end users in the course of measurement campaigns having pre-determined campaign objectives.

Inventors:
ROBITZA WERNER (AT)
DETHOF ALEXANDER (DE)
RAAKE PROF DR -ING (DE)
GÖRING STEVE (DE)
POLZEHL DR -ING (DE)
BEYER ANDRÉ (DE)
VON DEM BERGE RALF (DE)
HARTLEB FRANZ (DE)
Application Number:
PCT/EP2020/069783
Publication Date:
January 21, 2021
Filing Date:
July 13, 2020
Export Citation:
Click for automatic bibliography generation   Help
Assignee:
AVEQ GMBH (AT)
UNIV ILMENAU TECH (DE)
CROWDEE GMBH (DE)
DEUTSCHE TELEKOM AG (DE)
International Classes:
H04L12/24; H04L12/26
Foreign References:
US20180206136A12018-07-19
Other References:
HOSSFELD TOBIAS ET AL: "Best Practices for QoE Crowdtesting: QoE Assessment With Crowdsourcing", IEEE TRANSACTIONS ON MULTIMEDIA, IEEE SERVICE CENTER, US, vol. 16, no. 2, 1 February 2014 (2014-02-01), pages 541 - 558, XP011537238, ISSN: 1520-9210, [retrieved on 20140115], DOI: 10.1109/TMM.2013.2291663
KRISHNAN, S. S.SITARAMAN, R. K.: "Video stream quality impacts viewer behavior: Inferring causality using quasi-experimental designs", IEEE/ACM TRANSACTIONS ON NETWORKING, vol. 21, no. 6, 2013, pages 2001 - 2014, XP011534742, DOI: 10.1109/TNET.2013.2281542
DOBRIAN, F.SEKAR, V.AWAN, A.STOICA, I.JOSEPH, D.GANJAM, A.ZHANG, H.: "Understanding the impact of video quality on user engagement", ACM SIGCOMM COMPUTER COMMUNICATION REVIEW, vol. 41, no. 4, 2011, pages 362, XP058006665, DOI: 10.1145/2018436.2018478
HOSSFELD, T.KEIMEL, C.HIRTH, M.GARDLO, B.HABIGT, J.DIEPOLD, K.TRAN-GIA, P.: "Best practices for QoE crowdtesting: QoE assessment with crowdsourcing", IEEE TRANSACTIONS ON MULTIMEDIA, vol. 16, no. 2, 2014, pages 541 - 558, XP011537238, DOI: 10.1109/TMM.2013.2291663
HOWE, J.: "The rise of crowdsourcing", WIRED MAGAZINE, vol. 14, no. 6, 2006, pages 1 - 4
ROBITZA, W.AHMAD, A.KARA, P. A.ATZORI, L.MARTINI, M. G.RAAKE, A.SUN, L.: "Challenges of future multimedia QoE monitoring for internet service providers", MULTIMEDIA TOOLS AND APPLICATIONS, 2017
Attorney, Agent or Firm:
VOSSIUS & PARTNER (NO 31) (DE)
Download PDF:
Claims:
Claims

1. A system offered by a measurement system provider, for measuring the quality or other characteristics of one or more networks and/or one or more services that are transmitted over said networks by means of measurements performed by end users in the course of measurement campaigns having pre-determined campaign objectives, of the system comprising:

• a measurement system provider configured to carry out the measurement campaigns and to act as a crowd provider or commissioning one;

• a crowd provider configured to offer end users to perform measurements as crowd users in the context of crowd tasks;

• a plurality of end users for performing the measurements as measurement probe users, wherein the plurality of end users are users of the services or networks, each end user having a measurement probe allowing an end user to perform an active and/or passive measurement of the network;

• the measurement system provider and/or the crowd provider being configured to select the networks or services to be measured in a measurement campaign according to predetermined criteria;

• wherein the measurement probe is configured to generate measurement data, wherein the measurement data are technical parameters and/or context data, and/or user data, the technical parameters preferably being at least one of transfer rates, DNS resolution times, website load times, video load times, video bit rates, web URLs and domains, network requests;

• wherein the measurement probe is configured to perform at least one active or passive measurement, wherein the active measurement is started automatically at the request of the end user or on the basis of a predetermined time schedule and generates an active measurement environment which contains instrumentation or automation, to display the progress in the measurement and the measurement results, wherein the passive measurement takes place in the background without explicit interaction of the end user with the measurement probe, when one or more services are used by the end user;

• the system further comprising an evaluation unit configured to evaluate the measurement data by analyzing the collected technical parameters and/or context data and/or user data, including assessment of the network or service performance in the form of key performance indicators and evaluation of the quality in terms of the quality theoretically perceived by the user during a measurement, as determined in terms of the "Quality of Experience" using model algorithms.

2. The system of claim 1, wherein the crowd provider is configured to select and recruit the end users as crowd users.

3. The system of claim 2, wherein the crowd provider is configured to select and recruit the plurality of end users within the measurement campaign according to various dimensions by representative statistical sampling or targeted addressing.

4. The system of claim 1, 2, or 3, further comprising a verification unit for verification of the correct performance of crowd tasks and/or active measurements by matching and/or comparing the data collected in the tasks and/or measurements.

5. The system of claim 4, wherein verification takes place promptly and automatically after an active measurement.

6. A system according to any one of claims 1 to 5, wherein the system is further configured for re-tasking or future banning of end users to crowd tasks based on the output of the verification of past measurements.

7. A system according to any one of claims 1 to 6, wherein the measurement probe is configured to start an active measurement by means of specially formatted internet links (URLs), which are displayed in a crowd task and which contain the parameters for the measurement to be started.

8. A system according to any one of claims 1 to 7, further configured to transmit the data from the crowd tasks and/or measurement data to one or more databases.

9. A system according to any one of claims 1 to 8, further configured for mapping the crowd users and measurement probe users by exchanging identifiers, IDs, so that a unique relationship can be established between crowd users and end users using the measurement probe.

10. A system according to any one of claims 1 to 9, wherein the active measurement environment calls up one or more services and thereby simulates the behavior of an end user.

11. A system according to any one of claims 1 to 10, wherein the measurement probe is configured to perform an Internet speed test in the active measurement environment or passive measurement environment.

12. A system according to claim 10 or 11, wherein the measurement probe is configured to prevent disturbing factors in the active measurement environment which could negatively influence a measurement or make it invalid, disturbing factors preferably including parallel Internet traffic or the undesired interaction of the end user with the active measurement environment, and the prevention of these disturbing factors is achieved, preferably, by displaying notices to the end user, closing windows, or aborting ongoing downloads.

13. A system according to any one of claims l to 12, wherein the filtering of the measurement data is based on incongruities between individual data sets, abusive user behavior, or statistical methods such as percentile or inter-quartile filtering of measurement data;

14. A system according to any one of claims 1 to 13, the system further configured to determine the geolocation of the end user by acquiring technical parameters such as IP address or geolocation data of the device, including determination of the location or region and determination of the accuracy of the acquired geolocation.

15. A system according to any one of claims 1 to 14, the system further configured to determine the Internet Service Provider by applying a heuristic or other algorithmic method, which combines one or more of the user's information about their Internet Service Provider with other recorded technical parameters and/or context data and thereby calculates an "assumed Internet Service Provider (ISP)" per measurement date and/or per end user and optionally a certainty value for this determination.

16. A system according to any one of claims 1 to 15; wherein the evaluation unit is further configured to, when evaluating the quality of video services, transform events recorded by the measurement probe into video states by means of a state machine.

17. A system according to any one of claims 1 to 16, wherein the evaluation unit is further configured to evaluate the availability of a service or the network connection in several dimensions.

18. A system according to any one of claims 1 to 17, in which the measurement data are presented to the end user, preferably directly after an active measurement or by the user accessing a database.

19. A system according to any one of claim 1 to 18, wherein the measurement data is prepared by combining several individual measurement data and applying mathematical, statistical or graphic methods which serve the campaign objective.

20. A system according to claim 19, wherein the campaign objective comprises overall evaluation of an over-the-top service on the basis of several individual measurements, and/or statistical comparison of several providers.

21. A system according to any one of claims 1 to 20, wherein the measurement data is made available to third parties, preferably Internet service providers, video streaming providers, regulatory authorities, municipalities, or other infrastructure providers.

22. A system according to any one of claims 1 to 21, wherein the network is a fixed network, or mobile Internet.

23. A system according to any one of claims 1 to 22, wherein selection of the end users within the measurement campaigns is performed according to at least one of time, location, network services used, Internet connection types, device types, usage behavior, browser software, operating system.

24. A system according to any one of claims 1 to 23, wherein the crowd tasks include a query of technical parameters and/or user data and/or context data, and which include a task for the crowd user to initiate a measurement probe and perform a measurement with it.

25. A method for measuring the quality or other characteristics of one or more networks and/or one or more services that are transmitted over such networks by means of measurements performed by end users in the course of measurement campaigns having predetermined campaign objectives, the method being operated in a system comprising:

a measurement system provider for carrying out the measurement campaigns who acts as a crowd provider or commissions one;

a crowd provider configured to offer end users to perform measurements as crowd users in the context of crowd tasks; and

a plurality of end users for performing the measurements as measurement probe users, wherein the end users are users of the services or networks,

each end user having a measurement probe allowing an end user to perform an active and/or passive measurement of the network;

the method comprising the steps of:

selecting of the networks or services to be measured in a measurement campaign according to predetermined criteria;

performing measurement, by the measurement probe, that generates measurement data, wherein the measurement data are technical parameters and/or context data, and/or user data;

the technical parameters preferably being at least one of transfer rates, DNS resolution times, website load times, video load times, video bit rates, web URLs and domains, network requests;

wherein the measurement is at least one active or passive measurement, whereby the active measurement is started automatically at the request of the end user or on the basis of a predetermined time schedule and generates an active measurement environment which contains instrumentation or automation, displays the progress in the measurement and the measurement results; wherein the passive measurement takes place in the background without explicit interaction of the end user with the measurement probe, when one or more services are used by the end user;

evaluating the measurement data by analyzing the collected technical parameters and/or context data and/or user data, including assessment of the network or service performance in the form of key performance indicators and evaluation of the quality theoretically perceived by the user during a measurement, as determined in terms of the "Quality of Experience" using model algorithms.

26. The method of claim 25, further comprising the step of selecting and recruiting the end users as crowd users.

27. The method of claim 26, further comprising the step of selecting and recruiting the plurality of end users within the measurement campaign according to various dimensions by representative statistical sampling or targeted addressing.

28. The method of claim 25, 26, or 27, further comprising the step of verifying the correct performance of crowd tasks and/or active measurements by matching and/or comparing the data collected in the tasks and/or measurements.

29. The method of any one of claims 27 to 28, wherein verification takes place promptly and automatically after an active measurement.

30. The method of claim 29, further comprising re-tasking or future banning of end users to crowd tasks based on the output of the verification of past measurements.

31. A method according to any one of claims 25 to 30, further comprising starting an active measurement by means of specially formatted internet links (URLs), which are displayed in a crowd task and which contain the parameters for the measurement to be started.

32. A method according to any one of claims 25 to 31, further comprising transmitting the data from the crowd tasks and/or measurement data to one or more databases.

33. A method according to any one of claims 25 to 32, further comprising mapping the crowd users and measurement probe users by exchanging identifiers, IDs, so that a unique relationship can be established between crowd users and end users using the measurement probe.

34. A method according to any one of claims 25 to 33, in which the active measurement environment calls up one or more services and thereby simulates the behavior of an end user.

35. A method according to claim 34, wherein an Internet speed test is performed in the active measurement environment.

36. A method according to claim 34 or 35, further comprising preventing disturbing factors in the active measurement environment which could negatively influence a measurement or make it invalid, disturbing factors preferably including parallel Internet traffic or the undesired interaction of the end user with the active measurement environment, and the prevention of these disturbing factors is achieved, preferably, by displaying notices to the end user, closing windows, or aborting ongoing downloads.

37. A method according to any one of claims 25 to 36, in which the filtering of the measurement data is based on incongruities between individual data sets, abusive user behavior, or statistical methods such as percentile or inter-quartile filtering of measurement data.

38. A method according to any one of claims 25 to 38, further comprising determining the geolocation by acquiring technical parameters such as IP address or geolocation data of the device, including determination of the location or region and determination of the accuracy of the acquired geolocation.

39. A method according to any one of claims 25 to 39, further comprising determining the Internet Service Provider by applying a heuristic or other algorithmic method, which combines one or more of the user's information about their Internet Service Provider with other recorded technical parameters and/or context data and thereby calculates an "assumed Internet Service Provider (ISP)" per measurement date and/or per end user and optionally a certainty value for this determination.

40. A method according to any one of claims 25 to 39, wherein, when evaluating the quality of video services, the individual events recorded by the measurement probe are transformed into video states by means of a state machine.

41. A method according to any one of claims 25 to 40, wherein the availability of a service or the network connection is evaluated in several dimensions.

42. A method according to any one of claims 25 to 41, wherein the measurement data are presented to the end user, preferably directly after an active measurement or by the user accessing a database. 43. A method according to any one of the claims 25 to 42, wherein the measurement data is prepared by combining several individual measurement data and applying mathematical, statistical or graphic methods which serve the campaign objective.

44. A method according to any one of claims 25 to 43, wherein the measurement data is made available to third parties, preferably Internet service providers, video streaming providers, regulatory authorities, municipalities, or other infrastructure providers.

Description:
System and Method for Estimating the Quality of Networks or Services through

Crowdsourcing

Background of the Invention

Video streaming is one of the main drivers of current Internet traffic. As the delivered streaming quality continues to improve - also considering technological developments like 4K/UHD and higher, HDR and high framerate content - customer demands increase simultaneously. Also, web browsing is one of the most commonly used applications of the Internet for customers at home and on the go.

Studies indicate that users with faster connections and better services have higher expectations and may be disappointed faster in case of service problems (Krishnan, 2013). Considering this, video streaming over-the-top (OTT) providers, website providers, and Internet Service Providers (ISPs) generally optimize their services to increase network and service performance, and Quality of Experience (QoE) for their users.

Consequently, all providers along the service chain are incentivized to monitor and manage network and service performance and quality to keep customer experience high, facing challenges in this process (Robitza, 2017).

The quality of video streaming or web browsing (or other services) may be estimated based on simple bandwidth-related quality models, usually implemented via automated measurement or monitoring probes that regularly measure Key Performance Indicators (KPIs), informing about Quality of Service (QoS). However, these methods can only provide individual samples - usually according to a fixed time schedule - and do not necessarily reflect what is happening at the customer side. Also, Quality of Experience (QoE) models calculated via respective model algorithms would yield a better view on the user experience than merely relying on KPIs and QoS, which provide a purely technical perspective.

Crowdsourcing has been recently established as a valid means to "obtain the needed service by a large group of people, most probably an on-line community." (c.f. ITU-T Rec. P.912). By replacing automated measurement probes in fixed locations with crowd users performing measurement tasks, a much broader range of the population can be reached, yielding more valid and representative measurement results, particularly in the context of larger campaigns that aim to identify the network or service performance of, for example, different ISPs or services.

Hence, a crowdsourcing approach combined with the use of QoE models is a viable alternative to automated probing systems relying on QoS, with the benefit of providing large-scale longitudinal measures of real customer experience.

The term crowdsourcing has been publicized already in 2006 in an article by Howe (Howe, 2006) and used very generally. Since then, crowdsourcing has been increasingly used in the context of industrial and academic research, where the benefits of crowdsourcing were seen in the time and cost-efficient nature compared to performing laboratory studies with test participants.

In studies such as the ones from Krishnan et al. (Krishnan, 2013) or Dobrian et al. (Dobrian, 2013), data from real users was used to analyze user engagement and video streaming KPIs in the context of video streaming, however, the users were not part of a crowdsourcing campaign, or were actively asked to perform measurements; instead, their data was passively monitored without their explicit knowledge. Also, only performance was characterized in typical video streaming KPIs such as video loading time, whereas aspects of QoE were not considered.

Hossfeld et al. (Hossfeld, 2014) present an overview of best practices surrounding crowdsourcing in the context of QoE assessment, focusing on the case that users provide ratings of QoE during corresponding crowdsourcing tests. They mention the use of dedicated applications for showing video in a crowdsourcing task, highlighting requirements for presenting videos to the crowd users in a crowdsourcing context, and asking crowd users for their subjective feedback on the presented stimuli. However, these approaches do not consider measuring real OTT video services, instead relying on artificially generated stimuli or user interfaces being presented to the crowd users. Further, such crowdsourcing QoE studies typically require ratings from users, instead of carrying out a model-based assessment. Hence, the presented approaches cannot be used for monitoring of a real OTT service.

Overview of the Invention

The present invention is directed to a system and a method (in the following: system) which allows to determine the performance and end-user perceived quality of a network or of services which can be transmitted over such a network ("Over the Top"/OTT services), using crowd users to instantiate a number of measurement probes. Examples of the services addressed are video streaming or web browsing, but other services such as voice over IP (VoIP) or videoconferencing may be measured as well. These services may be transmitted over fixed-line Internet service, or a mobile network service.

The system can also be employed to generate other statements about the network or services, e.g. about the regional or temporal availability of services or the network.

The system is provided by a measurement system provider.

The system, among other properties, enables users with an Internet-capable device to measure the connection performance and/or quality of their Internet connection, and/or the performance/quality of services being transmitted over that connection, via measurements. The measurements can be performed in the context of measurement campaigns but can also be performed by the users in other contexts.

In the context of the present invention, measurement campaign means at least the provisioning (i.e., making available) of the system in which the measurements are to be performed by the end users, and the definition of campaign goals, which at least include a selection of the services to be measured.

The measurement system provider selects and preferably recruits users of the services to be measured via a crowd acquisition platform, at which point the users are henceforth described as crowd users. The measurement system provider can also employ/commission a third party to select end recruit the crowd users. The selection of crowd users can be made according to different dimensions, e.g. temporal or location-based, and under different statistical criteria in order to achieve representativeness of the selected panel.

Once recruited, the crowd users are given crowd tasks to perform. These tasks may comprise the gathering of technical parameters and asking users for user data (e.g. socio-economic status) or context data (e.g. information about their Internet access). The tasks may include instructions to initiate a measurement probe.

The crowd users initiate a measurement probe on their device. At this point the crowd users are called measurement probe users. The measurement probe can then be used to start a measurement. The measurement can be performed as soon as the user asks the probe to do so ("active" measurement), or it may be performed automatically and without explicit interaction (e.g. based on a time schedule; "passive" measurement). A measurement generates measurement data consisting of at least technical parameters, and optionally context data, and/or user data. Technical parameters consist of, among others, technical network parameters (e.g. signal strength in mobile networks), application transmission rates (e.g. upload and download speeds) or latency indicators (ping times), or application-specific events, or metadata (e.g. loading times of web pages and video services). These data are later analyzed with respect to determining the network or service performance or quality.

To analyze the data, the collected measurement data can be transformed or filtered, e.g. by linking several technical parameters collected by the measurement probe from different levels (e.g. events from the video player, data from network requests, data from a proprietary video player interface). Data can be transformed by enrichment (generating or including additional information from other data sources) with respect to the service or network provider, or the geolocation. Data can be filtered to enhance its validity.

Based on the measurement data, the network or service performance is characterized with Key Performance Indicators (KPIs). Based on the KPIs and other measurement data, the quality actually experienced by the user can be calculated in terms of the "Quality of Experience" via model algorithms.

Quality of Experience here is an estimation of user perception based on algorithmic models, which combines the collected measurement data and presents them in a simple, understandable form (e.g. as a school-grade-type scale). This modelling in connection with the actual technical data represents a novelty in connection with the overall system for measurement. The models to be used can be developed specifically for certain services.

The tasks conducted by the crowd users can be verified by comparing or checking the data collected in the crowd tasks with the data collected in the measurements performed by the user.

The gist of the invention lies in the unique combination of the aspects of the measurement system, by combiningthe use of crowd users to perform measurements of technical parameters in the context of a measurement campaign, as well as the evaluation of the Quality of Experience. According to a first aspect, the invention provides a system offered by a measurement system provider, for measuring the quality or other characteristics of one or more networks and/or one or more services that are transmitted over said networks by means of measurements performed by end users in the course of measurement campaigns having pre-determined campaign objectives, of the system comprising:

• a measurement system provider configured to carry out the measurement campaigns and to act as a crowd provider or commissioning one;

• a crowd provider configured to offer end users to perform measurements as crowd users in the context of crowd tasks;

• a plurality of end users for performing the measurements as measurement probe users, wherein the plurality of end users are users of the services or networks, each end user having a measurement probe allowing an end user to perform an active and/or passive measurement of the network;

• the measurement system provider and/or the crowd provider being configured to select the networks or services to be measured in a measurement campaign according to predetermined criteria;

• wherein the measurement probe is configured to generate measurement data, wherein the measurement data are technical parameters and/or context data, and/or user data, the technical parameters preferably being at least one of transfer rates, DNS resolution times, website load times, video load times, video bit rates, web URLs and domains, network requests;

• wherein the measurement probe is configured to perform at least one active or passive measurement, wherein the active measurement is started automatically at the request of the end user or on the basis of a predetermined time schedule and generates an active measurement environment which contains instrumentation or automation, to display the progress in the measurement and the measurement results, wherein the passive measurement takes place in the background without explicit interaction of the end user with the measurement probe, when one or more services are used by the end user;

• the system further comprising an evaluation unit configured to evaluate the measurement data by analyzing the collected technical parameters and/or context data and/or user data, including assessment of the network or service performance in the form of key performance indicators and evaluation of the quality in terms of the quality theoretically perceived by the user during a measurement, as determined in terms of the "Quality of Experience" using model algorithms.

According to a preferred aspect, the crowd provider is configured to select and recruit the end users as crowd users.

According to a preferred aspect, the crowd provider is configured to select and recruit the plurality of end users within the measurement campaign according to various dimensions by representative statistical sampling or targeted addressing.

According to a preferred aspect, the system further comprises a verification unit for verification of the correct performance of crowd tasks and/or active measurements by matching and/or comparing the data collected in the tasks and/or measurements.

According to a preferred aspect, verification takes place promptly and automatically after an active measurement.

According to a preferred aspect, the system is further configured for re-tasking or future banning of end users to crowd tasks based on the output of the verification of past measurements.

According to a preferred aspect, the measurement probe is configured to start an active measurement by means of specially formatted internet links (URLs), which are displayed in a crowd task and which contain the parameters for the measurement to be started.

According to a preferred aspect, the system is further configured to transmit the data from the crowd tasks and/or measurement data to one or more databases.

According to a preferred aspect, the system is further configured for mapping the crowd users and measurement probe users by exchanging identifiers, IDs, so that a unique relationship can be established between crowd users and end users using the measurement probe. According to a preferred aspect, the active measurement environment calls up one or more services and thereby simulates the behavior of an end user.

According to a preferred aspect, the measurement probe is configured to perform an Internet speed test in the active measurement environment.

According to a preferred aspect, the measurement probe is configured to prevent disturbing factors in the active measurement environment which could negatively influence a measurement or make it invalid, disturbing factors preferably including parallel Internet traffic or the undesired interaction of the end user with the active measurement environment, and the prevention of these disturbing factors is achieved, preferably, by displaying notices to the end user, closing windows, or aborting ongoing downloads.

According to a preferred aspect, the filtering of the measurement data is based on incongruities between individual data sets, abusive user behavior, or statistical methods such as percentile or inter-quartile filtering of measurement data;

According to a preferred aspect, the system is further configured to determine the geolocation of the end user by acquiring technical parameters such as IP address or geolocation data of the device, including determination of the location or region and determination of the accuracy of the acquired geolocation.

According to a preferred aspect, the system is further configured to determine the Internet Service Provider by applying a heuristic or other algorithmic method, which combines one or more of the user's information about their Internet Service Provider with other recorded technical parameters and/or context data and thereby calculates an "assumed Internet Service Provider (ISP)" per measurement date and/or per end user and optionally a certainty value for this determination.

According to a preferred aspect, the evaluation unit is further configured to, when evaluating the quality of video services, transform events recorded by the measurement probe into video states by means of a state machine. According to a preferred aspect, the evaluation unit is further configured to evaluate the availability of a service or the network connection in several dimensions.

According to a preferred aspect, the measurement data are presented to the end user, preferably directly after an active measurement or by the user accessing a database.

According to a preferred aspect, the measurement data is prepared by combining several individual measurement data and applying mathematical, statistical or graphic methods which serve the campaign objective.

According to a preferred aspect, the campaign objective comprises overall evaluation of an over-the-top service on the basis of several individual measurements, and/or statistical comparison of several providers.

According to a preferred aspect, the measurement data is made available to third parties, preferably Internet service providers, video streaming providers, regulatory authorities, municipalities, or other infrastructure providers.

According to a preferred aspect, the network is a fixed network, or mobile Internet.

According to a preferred aspect, selection of the end users within the measurement campaigns is performed according to at least one of time, location, network services used, Internet connection types, device types, usage behavior, browser software, operating system.

According to a preferred aspect, the crowd tasks include a query of technical parameters and/or user data and/or context data, and which include a task for the crowd user to initiate a measurement probe and perform a measurement with it.

According to another aspect, the invention provides a method for measuring the quality or other characteristics of one or more networks and/or one or more services that are transmitted over such networks by means of measurements performed by end users in the course of measurement campaigns having pre-determined campaign objectives, the method being operated in a system comprising:

a measurement system provider for carrying out the measurement campaigns who acts as a crowd provider or commissions one;

a crowd provider configured to offer end users to perform measurements as crowd users in the context of crowd tasks; and

a plurality of end users for performing the measurements as measurement probe users, wherein the end users are users of the services or networks,

each end user having a measurement probe allowing an end user to perform an active and/or passive measurement of the network;

the method comprising the steps of:

selecting of the networks or services to be measured in a measurement campaign according to predetermined criteria;

performing measurement, by the measurement probe, that generates measurement data, wherein the measurement data are technical parameters and/or context data, and/or user data;

the technical parameters preferably being at least one of transfer rates, DNS resolution times, website load times, video load times, video bit rates, web URLs and domains, network requests;

wherein the measurement is at least one active or passive measurement, whereby the active measurement is started automatically at the request of the end user or on the basis of a predetermined time schedule and generates an active measurement environment which contains instrumentation or automation, displays the progress in the measurement and the measurement results; wherein the passive measurement takes place in the background without explicit interaction of the end user with the measurement probe, when one or more services are used by the end user;

evaluating the measurement data by analyzing the collected technical parameters and/or context data and/or user data, including assessment of the network or service performance in the form of key performance indicators and evaluation of the quality theoretically perceived by the user during a measurement, as determined in terms of the "Quality of Experience" using model algorithms. According to a preferred aspect, the method further comprises a step of selecting and recruiting the end users as crowd users.

According to a preferred aspect, the method further comprising a step of selecting and recruiting the plurality of end users within the measurement campaign according to various dimensions by representative statistical sampling or targeted addressing.

According to a preferred aspect, the method further comprises a step of verifying the correct performance of crowd tasks and/or active measurements by matching and/or comparing the data collected in the tasks and/or measurements.

According to a preferred aspect, verification takes place promptly and automatically after an active measurement.

According to a preferred aspect, the method further comprises re-tasking or future banning of end users to crowd tasks based on the output of the verification of past measurements.

According to a preferred aspect, the method further comprises starting an active measurement by means of specially formatted internet links (URLs), which are displayed in a crowd task and which contain the parameters for the measurement to be started.

According to a preferred aspect, the method further comprises transmitting the data from the crowd tasks and/or measurement data to one or more databases.

According to a preferred aspect, the method further comprises mapping the crowd users and measurement probe users by exchanging identifiers, IDs, so that a unique relationship can be established between crowd users and end users using the measurement probe.

According to a preferred aspect, the active measurement environment calls up one or more services and thereby simulates the behavior of an end user.

According to a preferred aspect, an Internet speed test is performed in the active measurement environment. According to a preferred aspect, the method further comprises preventing disturbing factors in the active measurement environment which could negatively influence a measurement or make it invalid, disturbing factors preferably including parallel Internet traffic or the undesired interaction of the end user with the active measurement environment, and the prevention of these disturbing factors is achieved, preferably, by displaying notices to the end user, closing windows, or aborting ongoing downloads.

According to a preferred aspect, the filtering of the measurement data is based on incongruities between individual data sets, abusive user behavior, or statistical methods such as percentile or inter-quartile filtering of measurement data.

According to a preferred aspect, the method further comprises determining the geolocation by acquiring technical parameters such as IP address or geolocation data of the device, including determination of the location or region and determination of the accuracy of the acquired geolocation.

According to a preferred aspect, the method further comprises determining the Internet Service Provider by applying a heuristic or other algorithmic method, which combines one or more of the user's information about their Internet Service Provider with other recorded technical parameters and/or context data and thereby calculates an "assumed Internet Service Provider (ISP)" per measurement date and/or per end user and optionally a certainty value for this determination.

According to a preferred aspect, the method further comprises, when evaluating the quality of video services, the individual events recorded by the measurement probe are transformed into video states by means of a state machine.

According to a preferred aspect, the availability of a service or the network connection is evaluated in several dimensions. According to a preferred aspect, the measurement data are presented to the end user, preferably directly after an active measurement or by the user accessing a database.

According to a preferred aspect, the measurement data is prepared by combining several individual measurement data and applying mathematical, statistical or graphic methods which serve the campaign objective.

According to a preferred aspect, the measurement data is made available to third parties, preferably Internet service providers, video streaming providers, regulatory authorities, municipalities, or other infrastructure providers.

A further, and independent aspect of the present invention provides a method of estimating geolocation of a user. However, this aspect is also applicable to the aspects described above with all its embodiments.

The method of estimating the location of the user conducting a crowd task and/or performing one or more measurements comprises the following steps: acquiring first geolocation data associated with the ZIP code or hometown of the user; acquiring second geolocation data representing the geolocation during conduction of the crowd task by the user, according to a browser geolocation API or device API; acquiring third geolocation data representing the geolocation during instantiation of the measurement probe by the user, according to a browser geolocation API or device API; and acquiring fourth geolocation data while performing active or passive measurements by a measurement probe, the fourth geolocation data representing the IP address, or a browser geolocation API or device API of the user; and determining the geolocation of the user in terms of latitude/longitude pair based on acquired the first to fourth geolocation data.

The method preferably comprises a mapping of ZIP codes to latitude/longitude pairs as input data. The method further preferably comprises the step of determining, for any latitude/longitude pair, the nearest place name as well as, more preferably, the corresponding population can be determined.

The method further preferably comprises the step of estimating the real location of the user based on a heuristic integrating the different sources of information gathered.

In the preferred embodiment, the real location of the user and, optionally, a type of region based on different sources of information about geolocation that have been collected in the measurement campaign.

The real location may correspond to the user's home or assumed place of conducting the measurements.

The desire for estimating the real location stems from the fact that each individual source of information may not be accurate enough, and that only the combination of multiple sources may yield an accurate estimation of the real location.

A further, independent aspect of the present invention relates to a system and method to determine the Internet Service Provider of an end user by applying a heuristic or other algorithmic method, which combines one or more of the user's information about their Internet Service Provider with other recorded technical parameters and/or context data and thereby calculates an "assumed Internet Service Provider (ISP)" per measurement date and/or per end user and optionally a certainty value for this determination.

This aspect is also applicable to the aspects described above with all their embodiments.

For data analysis purposes, it may be required to know about the user's real Internet Service Provider. However, ISPs may route their Internet access products through other ISPs' networks. For example, ISP "A" may obtain network access from ISP "B" and offer a product under its own name, which, when used by users of ISP "A" transmits data over the network of ISP "B". Thus, the ISP name of a user may differ from the users' real ISPs. For example, when ISP "A" routes its products through ISP "B"'s networks, an actual user of ISP "A" may have his/her IP address resolved to the name of ISP "B". Such a resolution mismatch may cause data inconsistencies and invalid results in later data analysis steps, in particular when determining the service quality or Quality of Experience for a given ISP, when this ISP's measurement data contain incorrectly mapped IP addresses. This problem is overcome by this aspect of the invention. In the following, the "ISP based on IP address" (i.e., the result of the IP address resolution or mapping table lookup) is defined as "ISP-IP". Hence, in this aspect, the real ISP of a user is estimated ("assumed ISP") by the following information, if available:

1. The user profile, as based on the selection that was made for the user. For instance, a user may have been selected for him/her having Internet access from a certain ISP based on previous knowledge.

2. The user's IP address while instantiating the measurement probe ("Instantiation ISP", ISP- IP)

3. The measurement probe user's IP address when performing a measurement in the measurement probe ("Measurement ISP", ISP-IP).

Based on these data (or any combination of those), the user's "Assumed ISP" is estimated according to a heuristic.

Preferably, a certainty value that characterizes how likely it is that the determined Assumed ISP is correct is included in the heuristic.

Preferably, the method comprises a step of determining the user's "Most Used ISP" and "Most Used Assumed ISP" based on the Measurement ISP. This calculation or determination is preferably performed cumulatively, for example on a day to day basis, to ensure that when evaluating data from the past, no future knowledge will be used. Also, preferably, the "Most Used Assumed ISP" is determined by choosing the most plausible combination.

Further preferably, the system or method according to this aspect determines the "Per- Measurement Assumed ISP" and its certainty. This is preferably performed by an algorithm based on the Assumed ISP, the Most Used ISP, the Measurement ISP. Preferably, the set of possible ISP values are first classified into whether the ISP's IP addresses can always be correctly resolved or may be incorrectly resolved (i.e., shown as another ISP), and - if the second case is true - whose ISP's network these addresses are routed through.

In more detail, the system or method first determines whether the Assumed ISP belongs to those ISPs whose IP addresses can be resolved correctly, and if so, the Per-Measurement Assumed ISP is equal to the Assumed ISP. If not, it depends on whetherthere are incongruencies between Most Used Assumed ISP and Assumed ISP (in which case the Per-Measurement Assumed ISP is simply the Measurement ISP), and if not, it depends on the Assumed ISP of the user, and which ISP-IPs are expected for that user to appear if a measurement really stems from their Assumed ISP.

The invention will be described with reference to the drawings in which:

Fig. 1 shows a general system overview example;

Fig. 2 illustrates an example flow chart from the perspective of the crowd user. The user is presented a job which is then accepted; after acceptance the user initiates the measurement probe and starts the measurement, which sends measurement data;

Fig. 3 shows state transitions for events measured in the context of video playbacks; and

Fig. 4 shows aggregation of performance/quality with different functions. The measurements for service X and Y of User 1 can be first aggregated via functions fs x and fs v followed by fu or simply fu.

Detailed Measurement System Description - instantiation with external user recruitment

1. Introduction

This section describes a preferred embodiment for the system.

In this case, a measurement system provider launches a measurement campaign to determine the quality of the following services:

• web browsing

• video streaming

A crowd acquisition platform is offered by the measurement system provider.

Crowd users perform active and passive measurements in web browsers on PCs or comparable devices via a measurement probe. For this exemplary case of fixed-network access, the probe is realized as a web browser extension. In other contexts, e.g. for mobile networks, the probe can be realized as a mobile application.

Measurement data about web browsing and video streaming are captured by the measurement probe, sent to the measurement system provider, and analyzed in terms of service quality and Quality of Experience. It is understood that a similar instantiation can be adopted also for e.g. mobile networks, other services, using different quality models and data analysis procedures than the ones described henceforth.

To realize the measurement campaigns in the preferred embodiment, at least the following steps are performed:

1. Crowd user selection and recruitment

2. Participation of crowd user in crowd task

3. Instantiation of measurement probe

4. Measurement of the chosen service(s)

5. Data analysis

In the following, further details of the preferred embodiment are described.

Figure 1 shows an overview of the preferred embodiment in terms of the steps carried out by each entity.

Figure 2 shows an exemplary process for the crowd user participating in crowd tasks and initiating the measurement probe.

2. Measurement System Provider and Campaign Goals

In a preferred embodiment, the system is instantiated within a measurement campaign that has the goal of measuring the quality of web browsing and video streaming over fixed-line Internet connections. Other services such as video conferencing, telephony or data transmissions are also envisaged by the present invention as alternatives that can be measured in a similar fashion.

The specific measurement campaign goals defined by the measurement system provider in this instantiation may be to:

1. estimate the Quality of Experience perceived by the end user for a single measurement and/or multiple measurements, and/or

2. determine the Quality of Experience for a given service, optionally for a particular Internet Service Provider, and/or 3. determine the internet Service Provider offering the highest overall Quality of Experience, or a ranking of Internet Service Providers based on their Quality of Experience

Other goals or combination of goals may be chosen in other embodiments of the invention. The services to be measured may be chosen according to predetermined criteria, such as the representativeness for all services a typical Internet customer may use, or the typical customer behavior in terms of service usage.

3. Crowd

3.1. Crowd Acquisition Platform

In the preferred embodiment, a crowd acquisition platform is hosted and managed by the measurement system provider. The platform serves to screen/select and recruit crowd users.

In other embodiments, the crowd provider and measurement system provider may be separate entities by means of the measurement system provider subcontracting or partnering with a crowd provider. 3.2. Crowd User Selection

In the preferred embodiment, to obtain a pool of users that can later be invited to actual measurement tasks (conducted in the context of a measurement campaign), the measurement system provider may first select individual candidate users based on whether they match a given profile of several conditions required for their later participation. The selection of users within the measurement campaigns is preferably made according to measurement campaign goals defined by the measurement system provider.

In particular, the selection may be performed according to at least of the following parameters:

- The user's device (e.g., PC, tablet, mobile phone)

- The user's Internet provider

- The user's Internet connection type (e.g., DSL, Cable, 4G/LTE), Internet connection at home (e.g., WiFi, Ethernet), Internet connection product details (e.g., upload/download speed) - The client software (e.g., browser type and version, operating system vendor and version) used

- The time of activity of the user

- The user's geolocation

Other selection parameters may be defined based on the measurement campaign goals. For example, one parameter may be the user's usage of Internet services to be measured, with the criterion that the user uses the respective service at least n times per time period (e.g., once per week).

In the preferred embodiment, for the case of measuring the quality of video streaming on desktop PCs, only users whose device is of the type "PC", and who use the online streaming platform(s) to be measured at least once per week may be selected.

3.3. Recruitment

Through the crowd acquisition platform, in the preferred embodiment, the crowd provider may approach selected users by inviting the users to perform crowd tasks. This is referred to as recruitment. The recruitment may be performed via different communication channels such as email or direct messaging.

Users may also perform crowd tasks without having been actively recruited.

Users can participate in the tasks by opening the crowd acquisition platform in a browser or mobile application or by other means of access.

Users may be invited to tasks on different days of the week in order to ensure a distribution of the weekdays at which measurements occur.

3.4. Measurement Probe Instantiation

In the preferred embodiment, the recruited crowd users must instantiate the measurement probe (c.f. section 4) as part of the tasks.

To instantiate the measurement probe on the crowd users' devices, in the preferred embodiment, users may be supplied a web link (URL) by which the measurement probe will be instantiated on their device. For example, such a URL may be: http://measurement-system-provider.com/instantiate_measureme nt_probe Upon following the link, the measurement probe will be instantiated on the user device.

In other embodiments, they may acquire the probe through other means such as browser web stores or mobile app stores.

3.5. Data Collection in Tasks

When participating in a task, in the preferred embodiment, the following data may be obtained from the crowd users by means of surveys or automated data collection:

- Technical parameters

o Device type (e.g., PC, tablet, mobile phone)

o Browser type and version

o Operating system vendor and version

User data (e.g. socio-economic data)

Context data

o Internet provider

o Internet connection type (e.g., DSL, Cable, 4G/LTE), Internet connection at home (WiFi, Ethernet), Internet connection product details (upload/download speed) o Geolocation, via manual data entry (country, hometown, ZIP code) or automated data collection

Other information may be queried depending on the goal of the campaign.

3.6. Task Participation

Crowd users may be expected to participate in such tasks only once, or repeatedly participate in a number of tasks. In the preferred embodiment, the campaign may rely on repetitive participation of crowd users in consecutive tasks over a certain period of time. Two types of tasks are hence available to crowd users:

• Initial tasks are offered to users who belong to the selection. These initial tasks introduce the user to the scope of the campaign and the possibility to participate in multiple tasks over a period of time.

• Re-targeted tasks: For users who have completed a first task, several follow-up tasks are presented in the following time. Other goals may be followed as well, e.g. specific targeting of particular crowd users for particular days of the week or using particular services to be measured.

3.7. Number of Crowd Users

In the preferred embodiment, due to the natural churn and loss of crowd users - which can be seen as test participants in a repetitive study design the number of initial crowd users needs to be high enough to prospectively cover the expected loss over time. Furthermore, new crowd users have to be selected to ensure continued participation.

The crowd provider needs to supply a constant stream of user-active measurements such that a sufficient number of active crowd users are involved. In this case, "active" refers to a crowd user having conducted at least one task with the measurement probe in a given period of time.

The number of crowd users required depends on the chosen data analysis methods, and in particular the algorithms used to determine the service quality or Quality of Experience.

Example: Number of Crowd Users dependent on statistical power

In particular, one may choose the number of crowd users in such a way that the probability of finding a statistically significant difference between independent factors is increased during data analysis.

For instance, assume that the service quality of two ISPs "A" and "B" is to be statistically compared on the basis of the mean quality score per ISP, QI A and QI B , for a given service.

The mean quality score per ISP Ql A> Qh may be defined as the mean over all mean quality scores of all crowd users selected for the ISPs "A" and "B", respectively, i.e. in this case:

where QU n is the mean quality score for user n of ISP "A", and N A are the number of crowd users for ISP "A".

The mean quality score for user n is:

where I n is the number of measurements for user n, and M n are the per-measurement quality scores for that user. Here, it is assumed that the number of crowd users is the same for both ISPs and that all other assumptions required for t-tests (e.g. normality) hold.

Assuming a two-sided independent sample t-test, the required number of crowd users can be determined via a power analysis procedure (see also Cohen, Jacob. Statistical power analysis for the behavioral sciences. Academic press, 2013).

If the true difference of quality scores Q A and Q B is 0.5, and the standard deviation is 0.5, the significance level is 0.05, and the targeted test power is 0.9, then the required number of crowd users is 22.02, which rounds up to 23. Hence, at least 23 crowd users are required for each ISP.

3.8. Verification

Once a crowd user has completed a task, it may be technically verified that the user fulfilled the task correctly. The verification may happen by matching and/or comparing the data collected in the crowd tasks and/or measurements conducted in the course of the task(s).

For instance, when the task includes a measurement of a video playback, it may be verified that the measurement probe has played back the video for a required amount of time.

4. Measurement probe

The quality of the chosen services is measured with a measurement probe.

The measurement probe is, for example, realized as a web browser extension. Based on the chosen services, in the preferred embodiment, it measures video streaming and web browsing quality.

The measurement probe may be realized in other technical forms such as mobile applications, website scripts running in the context of a browser, or standalone software running natively on a desktop or mobile operating system.

This section gives details on the measurement probe and its technical components. Multiple versions or instances of the measurement probe may be used simultaneously by a crowd user, and these versions or instances may exchange data among each other or execute functions on the respective other version or instance. 4.1. Measurement data

The measurement probe is equipped to measure technical parameters of online video streams and web browsing, but in other embodiments may also measure other services such as video conferencing, telephony or data transmissions.

The use of the measurement probe creates measurement data. Measurement data are described in Section 1.6.

4.2. Data Transmission

The measurement probe is equipped with a sending mechanism to send its captured data to one or more central database servers.

In the case of send or server failures, the measurement probe may repeatedly try to send an event to the server(s) within a given period of. time, continuously ramping up the sending time intervals, up to a predefined point in time after the last trial. This allows delivering data to the server(s) in case that at least one of the following events takes place: the crowd user experiences Internet connectivity issues

- the measurement probe or the device experiences software issues

the server is not able to process the request(s) originating from the measurement probe

- the user prematurely closes the measured application (e.g. browser) before all data was sent successfully

5. Active and Passive Measurements

If the measurement probe is successfully instantiated on the crowd user's device, it can perform active and/or passive measurements. In the preferred embodiment, it performs both types of measurements, but it may also only perform one type of measurement for a realization of the described system.

5.1. Active Measurements

The active measurement is started automatically at the request of the crowd user.

In the preferred embodiment, the active measurement may be started upon clicking a specially formatted web link (URL) that is shown to the crowd user in the crowd task. The link may contain the parameters needed for launching the required active measurement. Upon clicking the link, the active measurement is started. For example, the URL may be: http://example.com/launch_measurement?service=video

The above URL may then launch the active measurement of the video service in the measurement probe.

The active measurement may also be started by means of a button or other interface elements of the measurement probe.

An active measurement may also be started on the basis of a predetermined time schedule. 5.1.1. Active Measurement Environment

When launching an active measurement, the measurement probe creates an active measurement environment, which contains instrumentation (also called automation) for performing the measurement. Instrumentation means that no or minimal user interaction is required for the conduction of the measurement.

The active measurement environment may offer the following functionalities:

• Overview of measurement types (for different services) that can be started by the user;

• Presentation of guidance and instructions for the user during the conduction of a measurement, including showing the progress;

• Automation of an Internet speed test to determine the user's upload and download bandwidth as well as ping times (latency);

• Automated video playback for a predefined length on different video streaming services;

• Automated website loading for a list of URLs, including pre-caching of DNS entries for each affected domain;

• Ensuring a valid test environment for the time of the study (see "disturbance factors" below);

• Chaining of tests so that different types of measurements (e.g., speed tests, video playback, and web browsing tests) can be combined into one single measurement;

• Showing the measurement results to the user

In the preferred embodiment, for the goal of measuring the quality of web and video services, the active measurement environment may perform: an Internet speed test a measurement of one or more websites for determining web browsing quality a measurement of one or more video playbacks on one or more video sites for determining video quality

5.1.2. Disturbance Factors

During an active measurement, disturbance factors, which could negatively influence or invalidate a measurement, may be prevented by the active measurement environment. Disturbance factors depend on the device and context of the user, and include, for example, parallel Internet traffic generated in browser tabs or through downloads, or the undesired interaction of the user with the active measurement environment.

The prevention of these disturbance factors may be achieved by having the measurement probe perform actions (e.g., display a notice to the user warning them of their disturbance, close browser tabs that cause disturbance, or canceling any downloads running in the web browser).

In the preferred embodiment, the following checks or restrictions may be set by the measurement system within the active measurement environment:

• The user is asked to acknowledge that they are at home at the time of the measurement.

The answer to this question and the corresponding geolocation is logged. If the user does not acknowledge being at home, the check is failed.

• Browsing in other tabs or windows of the same browser is restricted. If the user tries to browse to another website, the check is failed.

• Downloads in the browser are restricted. If a download is started, the check is failed.

• Network requests originating from other tabs are restricted in order to prevent background traffic.

When measuring video quality in particular, the following restriction may be set:

• Seeking in a video is made impossible by blocking the events, and/or presenting an overlay over the actual video.

If the user violates any of these checks, the measurement may be aborted by the active measurement environment, and the user can be instructed to be more careful about not intervening during the measurement. When finishing or aborting the measurement, these checks may be disabled again, so that regular usage of the device on which the measurement probe is instantiated is possible.

Furthermore, the following actions may be carried out in the active measurement environment in order to improve the validity of the measurement data:

• The browser window is maximized so that a video can be downloaded in the highest possible resolution that the user would normally see.

• The tab playing a video will be put in the foreground so that the browser does not throttle the performance of the playout process.

• The user may be asked to sign into or sign out from the services to be measured, or may be signed in and/or out automatically by the measurement probe.

• The browser cache can be deleted before the measurement in order to make sure that precached data is not impacting website load times. The user can be informed about this fact before starting the measurement.

5.2. Passive Measurements

Passive measurements take place in the background without explicit interaction of the user, once a relevant video portal or website is launched by the user. In this case, the probe can monitor the technical parameters in a non-intrusive manner and in the background.

During passive measurements, the restrictions listed in the active measurement environment do not apply. However, information about the context of measurement can still be passively recorded by the probe and used for later validation of the measurement data.

This context information may be used for checks similar to the active measurement environment, such as:

• Checking the geolocation of the measurement against the user's previously identified home location

• Checking whether, and how many other browser tabs are opened

• Checking whether, and how many downloads are currently running

• Checking whether, and how many network requests are originating from other browser tabs

• Checking whether the user seeks in a video 6. Measurement Data

In the preferred embodiment, a measurement generates measurement data consisting of the following, where at least technical parameters are required:

- Technical parameters

User data

Context data

The data are detailed in the following sections.

6.1. Technical Parameters

This section describes the technical parameters which may be measured by the measurement probe in the preferred embodiment.

Technical parameters consist of at least one of:

Technical network parameters

- Application transmission rates

Device type

Browser type and version

Operating system vendor and version

- Application-specific events and states

Application-specific data and metadata

Technical parameters can be measured both at network and application level.

Which parameters are collected depends on the service to be measured. In the preferred embodiment, where video quality and web browsing quality are to be determined, technical parameters are at least measured in terms of video playback events and states as well as web events and states.

6.1.1. Technical Network Parameters

Examples of technical network parameters are: measured signal strength in mobile networks

measured WiFi signal strength in WiFi networks

round-trip-times or one-way latency of IP-routed packets 6.1.2. Application Transmission Rates

Examples of application transmission rates are: upload transmission rate

- download transmission rate

These transmission rates may be measured via TCP or UDP, or other application-level protocols.

6.1.3. Device Type

The device type identifies the device on which the measurement probe is instantiated. It may be, for example:

- PC

- Tablet

Mobile phone

6.1.4. Browser Type and Version

The operating system name and version are identified based on APIs that the measurement probe may call, such as the User Agent string within a browser, or an operating system API.

6.1.5. Operating System Vendor and Version

Based on the User Agent string of the browser, the type of browser and its version are identified, in case that a measurement probe running in a browser is used.

6.1.6. Application-specific events: Video Playback Events and States

This section describes the application-specific events captured in the case of measuring video playbacks.

In the preferred embodiment, the following video playback events are measured:

• Unstarted: Video is not yet started. This event is measured at time t 0 , where t 0 equals the time at which the user initiated the start of a video playback (e.g., by clicking a respective link or button, or by entering a respective URL).

• Stalling: Player is rebuffering and indicates stalling to the user (this is also the case for initial loading). Stalling events are measured at time s n where n = [0, ... , N S ] and N s is the number of stalling events.

• Playing: Player is playing the video. The events are measured at time p n where n =

[0, ... , N p ] and N p is the number of playing events. • Paused: User has paused the playback. The events are measured at time h n where n = [0, ... , N h ] and N h is the number of paused events.

• Seeking: User is seeking in the video. This is usually followed by stalling, in case that the user seeks to a region for which no video data has been buffered yet. The events are measured at time e n where n = [0, ... , N e ] and N e is the number of seeking events.

• Ended: Playback has ended (end of video reached). The event is measured at time t x .

According to the sequence of the above events, the player can be in one of the following states, where each state has a starting point and a duration:

• Unstarted: No information about playback state is available, as the website or video player has just loaded. The duration of the unstarted state corresponds to the time x 0 — t 0 where x 0 can be any event time, i.e. stalling, playing, paused, seeking, or ended.

• Stalling: Player has stalled due to empty buffer. The duration of the state corresponds to x n — s n where x n can be any event time that resolves the stalling state, i.e. playing, paused, seeking or ended. If the state previous to the stalling was seeking, a special state "stalling after seeking" is defined.

• Playing: Player is playing. The duration of the state corresponds to the time x n — p n where x n can be any event following a playing state, i.e. stalling, paused, seeking, or ended.

• Paused: Player is paused by the user. The duration of the state corresponds to the time x n — h n where x n can be any event following a paused state, i.e. playing, stalling, seeking, or ended.

• Seeking: Player is seeking to a certain position. The duration of the state corresponds to the time x n — e n where x n can be any event following a seeking state, i.e. stalling, playing, or ended.

The transition between states can be enabled by the events, e.g., using a state machine as shown in Fig. 3.

In case other events are measured in other embodiments, other states can be defined. For example, events related to video advertisements may be measured, and a corresponding "advertisement" state may be defined based on those. 6.1.7. Application specific Data and Metadata: Video Playback Data and Metadata

This section describes the application-specific data and metadata captured in the case of measuring video playbacks.

Data on:

• the video's coded image size

• the video element size (i.e. the dimensions of the video on the web page)

• the video bitrate

• the video codec used

• the video framerate used

• the audio codec used

• the audio bitrate used are gathered to be able to estimate the quality of the stream. Which data are collected depends on the quality algorithm chosen for estimating the quality.

The data may be sampled at specified intervals of time (e.g., every second) to ensure an accurate measurement.

Depending on the capabilities of the measurement probe, additional data may be gathered:

• the transmitted audio bitstream and/or decoded signal

• the transmitted video bitstream and/or decoded video pixels

• metadata or feature descriptions about the audio and/or video bitstreams and/or signals originating from the server from which the audio/video bitstreams were transmitted

• the original audio and/or video signal, before encoding them for HTTP Adaptive Streaming, from the server from which the audio/video bitstreams were transmitted

Further metadata may be queried from the video, such as:

• the video identifier on the video service platform

• the video content type

• the video author's description

• the video tags

• the video comments or the number thereof

• the number of video likes or dislikes • the number of video views

6.1.8. Web Events and States

For each relevant website loaded actively or passively, in the preferred embodiment, the measurement probe may collect the following events:

• Tab created: User creates a tab in the browser

• Tab closed: User closes the tab in the browser

• Document ready: An event which fires when the site has been "completely loaded and parsed, without waiting for stylesheets, images, and subframes to finish loading" as specified in the DOMContentLoaded event by the W3C Navigation Timing API, or similar events that may correspond to the same instance in time.

• Window load: The Javascript "load" event which fires when all resources have been loaded, or similar events that may correspond to the same instance in time

These events are used to later characterize the quality of the web session through its loading performance.

In another embodiment of the invention, the loading performance may also be characterized by other events such as the event when the website has reached visual completeness.

6.2. Context Data

Context data include:

Time

Internet service provider

Internet connection type

- Geolocation

6.2.1. Time

The timestamps of technical parameters collected in the measurement probe may be used as context data.

The timestamp of incoming requests at the central backend servers may also be used as context data. 6.2.2. Internet Service Provider

Based on the IP address of requests originating from the measurement software, the name of the Internet Service Provider and Autonomous System (AS) to which the IP address belongs can be resolved using external services or databases.

6.2.3. Internet Connection Type

The Internet connection type may be collected as context data, for example DSL, cable, 4G/LTE.

The Internet connection used at home (e.g., WiFi, Ethernet) may be collected.

Internet connection product details (contract upload/download speed) may be collected in addition.

6.2.4. Geolocation

In the preferred embodiment, where the measurement probe is instantiated in a browser, the user's geolocation may be queried from the web browser's API.

In other embodiments, the geolocation may be queried from the respective device or software APIs. Which method is used to determine the exact location depends on the device capabilities and the accuracy of the available location services for the user's particular position.

Typically, nearby Wi-Fi access points or built-in GPS receivers are used to determine the position and exposed through the browser API. In a mobile environment, mobile devices may use cellular info to determine the geolocation.

The user's IP address may also be used to determine their location.

The geolocation may also be queried via manual data entry (country, hometown, ZIP code).

7. Data Transmission and Storage

In the preferred embodiment of the invention, the measurement probe sends the measurement data to one or more central backend database servers that capture all data retrieved from the measurement probe. The data may be stored in a central relational or similar database.

8. Data Analysis

This section details the data analysis performed in the preferred embodiment. The data analysis is based on the measurement data sent from the measurement probe and its users and the data about crowd tasks that have been conducted by crowd users. The data analysis may happen at one or more central servers, but parts of the data analysis can also be performed already on the measurement probes themselves.

8.1. Transformation

Data may be transformed during analysis. In the preferred embodiment, transformation includes enrichment of data based on external sources of data (e.g. an ISP resolution platform or a video service provider).

Other data transformation steps may be required depending on the specific data layout or data analysis goals.

8.1.1. Crowd Data Merging

If the measurement system provider and the crowd provider are not the same entity, different database or storage systems may be used, which may use different sets of identifiers to identify the crowd users and instances of the measurement probe.

In this case, a merging (alternatively called "synchronization" or "mapping") of crowd user and measurement probe user identifiers is required, such that a relationship between crowd users and measurement probe users can be established during data analysis.

In particular, one crowd user may use multiple instances of measurement probes, and one instance of a measurement probe may be used by multiple crowd users at the same time.

The two sets of user identifiers may be defined as:

Ic = [ 1, 2. N c ]

1M = \A> B, , NM]

Where I c and I M are the user IDs and N c and N M are the number of crowd users and measurement probe users, respectively.

The sets can be referred to as having M:N cardinality.

The exchange of the crowd user and measurement probe user identifiers can be facilitated by the specially formatted web links used by crowd users during the conduction of their crowd tasks. For example, in the crowd task, the URL given by the crowd provider to the crowd user to instantiate the measurement probe may be: http://measurement-system- provider.com/instantiate_measurement_probe?crowd_user_id=A

In this case, the server operating behind the URL, owned by the measurement system provider, or alternative the measurement probe itself may store the ID of the crowd user that is attempting to instantiate the measurement probe ("A").

For each crowd user, during data transformation, the corresponding crowd task data from the crowd provider can then then merged with the data sent from the measurement probe(s) that the crowd user is/are/has been using. This includes all crowd users' completed tasks including their data from the crowd provider as well as all active and passive measurement data from the measurement probe(s).

To perform the merging, a mapping may be created between measurement probe user IDs and the crowd provider's user IDs. In the preferred embodiment of the invention, the merging may be performed by the measurement system provider based on the following process:

1. The crowd user ID is forwarded to the measurement probe during instantiation, and subsequently the measurement system provider, where it is stored.

2. A mapping table is created between the measurement probe user ID and the crowd user ID.

Example for Mapping

Assuming that in the above-given example URL, the crowd user with ID "A" instantiates measurement probe with ID "1", conducts at least one measurement, and then deletes the instance of the measurement probe, and the same crowd user then instantiates another instance of the measurement probe with the same crowd user ID, but the measurement probe now having ID "2", and conducts at least one measurement, then after the merging, the measurements from the measurement probes with IDs "1" and "2" both are linked to the crowd user with ID "A".

Likewise, if a measurement probe with ID "3" is instantiated by crowd users with the IDs "A" and "B", then measurements conducted with that measurement probe are linked to both crowd users. 8.1.2. Geolocation Estimation

In the preferred embodiment, it may be required to estimate the user's real location and, optionally, a type of region based on different sources of information about geolocation that have been collected in the measurement campaign.

The real location may correspond to the user's home or assumed place of conducting the measurements.

The necessity for estimating the real location stems from the fact that each individual source of information may not be accurate enough, and that only the combination of multiple sources may yield an accurate estimation of the real location.

The user's real location may be estimated based on the following different sources of geolocation information, if available:

1. The crowd user's ZIP code or hometown as entered in each crowd task;

2. The geolocation during conduction of the crowd task, according to a browser geolocation API or device API;

3. The geolocation during instantiation of the measurement probe, according to a browser geolocation API or device API;

4. The geolocation while performing active or passive measurements, according to the IP address, or a browser geolocation API or device API;

Based on entered ZIP codes or hometown, the geolocation (in terms of latitude/longitude pair) can be determined. This process is called "geocoding". A mapping of ZIP codes to latitude/longitude pairs is required as an input for this step.

Conversely, for any geolocation (i.e., latitude/longitude pair) that was collected, the nearest place name (city, county, state) as well as the corresponding population can be determined. This is called "reverse geocoding".

The user's real location may be estimated based on a heuristic integrating the different sources of information gathered.

Example: Algorithm for Estimating Real Location

For instance, the algorithm to determine the real location for each crowd user may be: 1. Use the ZIP code entered by the crowd user during the first crowd task; geocode it to a latitude/longitude pair

2. Store the result of Step 1 as "crowd user temporary home"

3. For each subsequent crowd task of that user, compare the geolocation against the "crowd user temporary home".

1. If the number of crowd tasks is larger than a predetermined number of tasks, and majority of crowd task geolocations corresponds to the "crowd user temporary home" within a certain accuracy, store the result as "crowd user home".

2. Otherwise, stop the algorithm.

4. For each measurement, compare the geolocation against the "crowd user home".

1. If the majority of measurement geolocations corresponds to the "crowd user home" within a certain accuracy, store the result as "user home".

2. Otherwise, stop the algorithm.

5. Output the "user home" as the user's real home location

For instance, the number of crowd tasks required in step 3.1. above may be 5, and the accuracy required for steps 3.1 and 4.1 may be 25 km. The values may be chosen differently according to the required precision of the algorithm.

If the algorithm has been stopped before the last step, this may be an indication of unreliable data which may not be used for further data analysis (see Sec. 8.2 below).

Mapping to Regions

Based on the population associated to a geolocation, the type of region may additionally be specified and used for further data analysis:

1 "Unknown" if population is unavailable

2 "City" if population 100,000 or higher

3 "Medium city" if population between 20,000 and 100,000

4 "Small city" if population between 5,000 and 20,000

5 "Rural" otherwise

Other mappings of population to region are possible. 8.1.3. Internet Service Provider Determination

For data analysis purposes, it may be required to know about the user's real Internet Service Provider, in particular if the goal of the measurement campaign includes characterizing the service quality or Quality of Experience of different ISPs.

Background: Routing and Wholesale

ISPs may route their Internet access products through other ISPs' networks - a process that may be referred to as "wholesale". For example, ISP "A" may obtain network access from ISP "B" and offer a product under its own name, which, when used by users of ISP "A" transmits data over the network of ISP "B".

IP resolution services or mapping tables may be used during data enrichment to query information about an IP address and resolve it to the name of an ISP who owns the IP address. Such a service or mapping table may be available by means of an Application Programming Interface or a database from where the information can be looked up.

For example, an IP resolution service or mapping table may offer an API available at a given URL, and it may return information on the ISP after requesting the URL.

For instance, the URL: http://ip-resolution-service.com/resolve?ip=1.2.3.4 may return information including the name of the ISP owning the IP address 1.2.3.4.

Such IP resolution services or mapping tables may show ISP names that differ from the users' real ISPs. For example, when ISP "A" routes its products through ISP "B'"s networks, an actual user of ISP "A" may have his/her IP address resolved to the name of ISP "B".

Such a resolution mismatch may cause data inconsistencies and invalid results in later data analysis steps, in particular when determining the service quality or Quality of Experience for a given ISP, when this ISP's measurement data contain incorrectly mapped IP addresses.

Information for ISP Determination

In the following, the "ISP based on IP address" (i.e., the result of the IP address resolution or mapping table lookup) is defined as "ISP-IP". Hence, in the preferred embodiment, the real ISP of a user may be estimated ("assumed ISP") by the following information, if available: 1. The crowd user profile, as based on the selection that was made for the crowd user ("Crowd Selection ISP"). For instance, a crowd user may have been selected for him/her having Internet access from a certain ISP based on previous knowledge.

2. The crowd user's answers to any question about their ISP, while performing the crowd tasks ("Crowd Task Answer").

3. The crowd user's IP address while instantiating the measurement probe ("Instantiation ISP", ISP-IP)

4. The measurement probe user's IP address when performing a measurement in the measurement probe ("Measurement ISP", ISP-IP). Determination of Assumed ISP

Based on information points 1., 2., and 3. (or any other combination of information such as the above), the user's "Assumed ISP" may be estimated according to a heuristic, including an optional certainty value that characterizes how likely it is that the determined Assumed ISP is correct. An example for a heuristic to estimate the Assumed ISP per user may be as follows, as given in pseudocode:

Here, "certainty" is the determined certainty value for the Assumed ISP being correct.

Dependent on whether the Assumed ISP is routing its networks through another ISP's network and hence is susceptible to its IP address being resolved incorrectly, there can be two results of the Assumed ISP determination:

1. The user has an Assumed ISP whose IP address is always correctly resolved

2. The user has an Assumed ISP whose IP address may be incorrectly resolved, i.e. shown as another ISP

As a result, for case 2, when the user was selected for using ISP "B" and claimed to be a customer of ISP "B", it may be determined that while the user's IP address during instantiation of the measurement probe stemmed from ISP„A", the user is actually a customer of ISP„B". In this case, ISP„B" is the Assumed ISP.

Determination of Most Used ISP and Most Used Assumed ISP

In addition to determining the Assumed ISP, the user's "Most Used ISP" and "Most Used Assumed ISP" may be determined based on the Measurement ISP.

For instance, if a user performed 10 measurements using ISP "A" and 15 measurements using ISP "B", the user's most used ISP is ISP "B". This calculation may be performed cumulatively, for example on a day to day basis, to ensure that when evaluating data from the past, no future knowledge will be used.

The Most Used ISP, since it is the result of an IP address resolution, may differ from the Assumed ISP due to the resolution reasons mentioned above. Hence, the "Most Used Assumed ISP" may be determined by choosing the most plausible combination. If, for example, the Most Used ISP of the user in the above example was ISP-IP„A", the Most Used Assumed ISP would actually be ISP„B".

Once this information is known, it can be expected that when the user produces a session with ISP-IP„A", it is actually ISP "B".

Determination of Per-Measurement Assumed ISP

The "Per-Measurement Assumed ISP" and its certainty may be determined by an algorithm based on:

- the Assumed ISP

the Most Used ISP

- the Measurement ISP

Here, the set of possible ISP values have to be first classified into whether the ISP's IP addresses can always be correctly resolved or may be incorrectly resolved (i.e., shown as another ISP), and - if the second case is true - whose ISP's network these addresses are routed through.

For instance, assume that the ISPs "A" and "B" always have their IP addresses correctly resolved, since they do not route their addresses through other ISP's networks. Further assume that ISPs "C", "D" and "E" may have their addresses incorrectly resolved, and that ISP "C" routes their networks only through ISP "A", and that ISPs "D" and "E" route their addresses through ISPs "A" or "C".

In the above case, the Assumed Per-Measurement ISP can be determined as follows:

The algorithm may be changed accordingly for different sets of ISPs.

It first determines whether the Assumed ISP belongs to those ISPs whose IP addresses can be resolved correctly, and if so, the Per-Measurement Assumed ISP is equal to the Assumed ISP. If not, if depends on whether there are incongruencies between Most Used Assumed ISP and Assumed ISP (in which case the Per-Measurement Assumed ISP is simply the Measurement ISP), and if not, it depends on the Assumed ISP of the user, and which ISP-IPs are expected for that user to appear if a measurement really stems from their Assumed ISP.

The certainty value for determining an Assumed Per-measurement ISP, i.e. the Per Measurement Certainty, may be calculated as follows:

Here, again, the algorithm may be changed accordingly for different sets of ISPs.

The certainty is based on the percentage of the observed ISP-IP in all of the user's measurements, called ISP Percentage. For example, if a user conducted 10 measurements from ISP "A" and 15 measurements from ISP "B", and the Assumed Per-measurement ISP for one

15

measurement is ISP "B", the ISP Percentage may be calculated as— - * 100 = 60. Any measurement where the Per-Measurement Assumed ISP is not equal to the assumed ISP may later be filtered out; as are measurements where the assumed measurement ISP differs from the most used ISP.

Finally, measurements where the assumed measurement ISP certainty is below a given percentage may be filtered out as well. This threshold may be variable depending on the capabilities for reliably identifying ISPs.

8.2. Filtering

Data may be filtered in order to enrich the overall quality or validity of the data used for later analysis. Filtering may happen temporarily while analyzing the data, or permanently, before or after data is stored.

In the preferred embodiment, the following filtering steps may be implemented, but it is understood that depending on the goal of the measurement campaign, different filters may be implemented, too.

8.2.1 Crowd Data

The following data related to crowd data may be filtered:

• Invalid data submitted by the crowd users may be filtered or corrected. Here, for example, context data about the geolocation with unknown or not properly formatted ZIP codes entered by the user may be removed.

• Assuming the exemplary case from Sec. 8.1.1, multiple crowd users may be using the same instance of a measurement probe. The data from measurement probe users for which there are multiple crowd users may be filtered, since it is ambiguous as to which crowd user's data should be further analyzed.

8.2.2. Measurement Data

The following data related to measurement data may be filtered:

• Wrongly formatted data sent by the measurement probe.

• Data from outdated measurement probe versions;

• Data from users who have already removed the measurement probe from their system;

• Data from users manually blocked by the measurement system provider;

• Data not originating from the measurement probe at all; • Any data that do not conform to the database input specification;

Data that indicate a wrong date or time set in the probe, which may lead to wrong timestamps (for, e.g., video playbacks or web sessions);

Further filtering may be implemented based on incongruities between individual data sets, or abusive user behavior.

Statistical methods such as percentile or inter-quartile filtering may be used in order to remove measurement data that lie outside the normal range.

8.3. Network or Service Performance Calculation

For each measurement, Key Performance Indicators (quality-relevant indicators) may be calculated based on the measurement data.

A list of possible KPIs for each type of service is given in the following sections and used in the preferred embodiment.

Based on the KPIs, the performance of one or more networks or service can be calculated using aggregation functions (see Sec. 8.4.2).

8.3.1. Video

The following KPIs may be calculated:

• Initial loading delay: The duration between the user's intent to load a video and the first successful video play event, as indicated by the player. It corresponds to the duration of the first stalling state.

• Total stalling time: Total duration of all stalling events in a playback. This does not include initial loading delay. It corresponds to the total duration of all stalling states.

• Average stalling time: Average length of all stalling events. This does not include initial loading delay. It corresponds to the average duration of all stalling states.

• Number of stalling events: The total number of stalling events.

• Longest played resolution: The resolution of the video that was used in the longest playing state.

• Initial resolution: The resolution of the video in the first playing state. 8.3.2. Web Browsing

The following KPIs may be calculated:

• Page load time: The time for loading the complete page. The time from navigationStart until loadEventEnd in the W3C Navigation Timing API can be used to determine this KPI.

• Document Ready load time: The time for loading the initial document, ignoring DNS resolution. The time from navigationStart until domContentLoadedEventEnd in the W3C Navigation Timing API can be used to determine this KPI.

8.4. Quality Calculation

For the measurements collected in the context of the measurement campaign, the quality experienced by the user may be calculated in terms of "Quality of Experience" via model algorithms (henceforth "quality").

This calculation is performed without requiring the user to vote or give feedback about the quality they actually experienced.

8.4.1. Background on Quality of Experience

In the context of this invention, Quality of Experience is defined as "the degree of delight or annoyance of the user of an application or service. It results from the fulfilment of his or her expectations with respect to the utility and / or enjoyment of the application or service in the light of the user's personality and current state." This definition is made according to ITU-T Rec. P.10.

Quality of Experience is different from Quality of Service in that it does not exclusively focus on objectively or instrumentally measurable technical parameters and related data, but also incorporates the human perspective in terms of how the services are experienced by users.

Model algorithms are algorithms that take as input different data of a measurement and output a Quality of Experience score. The Quality of Experience score may be given on a scale from 1- 5 labeled as Mean Opinion Score (MOS), where 1 corresponds to "Bad" and 5 to "Excellent". Other scales may be used as well. MOS values reflect the average rating that a group of users would give to a stimulus presented. In the context of this invention, MOS values may be calculated according to an algorithm that predicts these ratings.

The calculation can be performed at a central server or directly at the measurement probe. 8.4.2. Quality of Experience for Video

For video, in the preferred embodiment, any model that can use the measurement data can be employed to calculate a Quality of Experience score for a video playback.

The models according to ITU-T Rec. P.1203 and its further components P.1203.1, P.1203.2 and P.1203.3 can be employed to calculate an estimated value for the QoE of each video playback session of up to 5 minutes length. In particular, the algorithm according P.1203.1 mode 0, when used in combination with ITU-T Rec. P.1203.3 takes as input: audio/video bitrate, video resolution, frames per second, and stalling events (including initial loading) as well as information about the device used (e.g. PC, mobile phone).

In other embodiments, instead of the models from ITU-T P.1203, any other model that predicts Mean Opinion Scores (MOS) or similar ratings for the corresponding video service can be used, provided that they can operate on the recorded measurement data. For example, in case that the measurement data includes audio and/or video signals received at the measurement probe, "no-reference" algorithms can be used to predict audio and/or video signal quality, respectively. If further data on audio and/or video signals before encoding for FITTP Adaptive Streaming have been gathered, a "full reference" algorithm can be used.

8.5. Aggregation of Quality or Performance

Depending on the measurement goals, different aggregations may be performed on the service or network performance (see Sec. 8.3) or quality (see Sec. 8.4 scores.

In the preferred embodiment, it is assumed that KPIs or the quality via model algorithms can be determined for one measurement of one user using one particular service, with one ISP, resulting in the per-measurement score M x where x stands for the index of the measurement in the set of measurements M.

Simple Aggregation Functions

In the case of determining Quality of Experience for multiple measurements, multiple services, multiple users, or multiple ISPs, respective statistical aggregations may be performed based on functions applied to the per-measurement Quality of Experience scores. Similar aggregations can be performed on other independent variables present in the data, such as the user's region, the time of the day, or the day of the week.

Examples for aggregation functions are: fu for aggregating multiple measurements of one user, creating QU U ;

- f j for aggregating multiple measurements of one ISP, creating QI ; ;

- f s for aggregating multiple measurements of one service, creating QS s ; where u, i and s are the indices of the users, ISPs, or services, respectively.

In addition, multiple functions may be defined for each instance of user, ISP, or service, respectively, yielding fu u ,fi., or f Sg , respectively.

Fig. 4 shows an example of applying the functions f v and f s on multiple measurements of services X and Y made by two users 1 and 2.

In the simplest case, f Ut f and f T are defined as the mean over all scores M:

where t 6 [U, I, T] and N is, for the specific instantiation of the function, equal to the set of measurement score indices that correspond to all scores of the user, ISP, or service, respectively, which are to be analyzed.

Instead of the mean, other common aggregation functions like the median can be chosen. More complex functions may be chosen depending on the data analysis goals.

Composed Aggregation Functions

In a more advanced variant, the aggregation functions may be composed in a particular calling order, such that different functions are applied in a stepwise manner and create multiple levels of aggregation. For example:

- Per-service aggregation, followed by user aggregation: f u may be called after calling f Ss for each service in each user's measurement of services X and Y (e.g. S x and S Y ), in order to first aggregate measurements of the same service for one user, and then aggregate the results for the user, creating QS X S Y U U .

- Per-user aggregation, followed by ISP aggregation: /, may be called after calling f u for each user in an ISP's measurement data, in order to first aggregate measurements of one user, and then aggregate the results for the ISP, creating QU y f.

In the first example, it is assumed that the aggregation functions for measurements of a particular service differ from one another. Fig. 4 shows an example of this application, calling f Sx and f s for the measurements of services X and Y for one user, then applying f u to create QS X S Y U U .

The second example particularly differs from simply aggregating all measurements for one ISP without regard for the user. It yields a perspective on the ISP's quality where the number of measurements performed by each individual user has a lower impact on the overall aggregation result compared to aggregating over all measurements.

While the invention has been illustrated and described in detail in the drawings and foregoing description, such illustration and description are to be considered illustrative or exemplary and not restrictive. It will be understood that changes and modifications may be made by those of ordinary skill within the scope of the following claims. In particular, the present invention covers further embodiments with any combination of features from different embodiments described above and below.

Furthermore, in the claims the word "comprising" does not exclude other elements or steps, and the indefinite article "a" or "an" does not exclude a plurality. A single unit may fulfil the functions of several features recited in the claims. The terms "essentially", "about", "approximately" and the like in connection with an attribute or a value particularly also define exactly the attribute or exactly the value, respectively. Any reference signs in the claims should not be construed as limiting the scope.

References (the content of which is herein incorporated by reference)

Dobrian, F., Sekar, V., Awan, A., Stoica, I., Joseph, D., Ganjam, A., ... Zhang, H. (2011). Understanding the impact of video quality on user engagement. ACM SIGCOMM Computer Communication Review, 41(4), 362.

- Hossfeld, T., Keimel, C, Hirth, M., Gardlo, B., Habigt, J., Diepold, K., & Tran-Gia, P. (2014).

Best practices for QoE crowdtesting: QoE assessment with crowdsourcing. IEEE Transactions on Multimedia, 16(2), 541-558.

Howe, J. (2006). The rise of crowdsourcing. Wired magazine, 14(6), 1-4. Krishnan, S. S., & Sitaraman, R. K. (2013). Video stream quality impacts viewer behavior: Inferring causality using quasi-experimental designs. IEEE/ACM Transactions on Networking, 21(6), 2001-2014.

- Robitza, W., Ahmad, A., Kara, P. A., Atzori, L, Martini, M. G., Raake, A., & Sun, L (2017).

Challenges of future multimedia QoE monitoring for internet service providers.

Multimedia Tools and Applications.