August 2019
One year ago, Tutela introduced Consistent Quality as its new flagship metric to measure end-user Quality of Experience. Tutela was the first mobile network crowdsourced data company to introduce a metric that evaluates the quality of a connection against a range of thresholds, incorporating more than just download speed into the equation for what makes the “best” wireless network.
The response from industry has been emphatic: network operators, equipment vendors, and media outlets have cited our Consistent Quality findings in markets all around the world. Industry experts have been clear about the need for metrics that more accurately reflect Quality of Experience, and we’ve engaged with them to keep driving the conversation around quantifying end-user quality of experience.
Moreover, Tutela recognizes that there is a constant evolution in demands on wireless networks and what constitutes an adequate connection. We always strive to provide the best possible insights into end-user Quality of Experience, and we believe that updates we’re making to the Consistent Quality metric do just that.
From September 1st, Tutela will implement an updated standard for Excellent Consistent Quality measurements in all reports, and will introduce a new Core Consistent Quality standard specifically tailored to use-cases like standard-definition video streaming.
New Core Consistent Quality and updated Excellent thresholds
The “Excellent” quality throughput thresholds have been updated to reflect the latest available network performance recommendations from services like Netflix, Skype (including Skype for Business) and others, as well as emerging use cases such as live video streaming on services like YouTube and streaming video games. The most significant difference is the increase in download throughput requirement, from 4 Mbps to 5 Mbps, ensuring that “excellent” connections meet the requirements of most mobile video services for streaming 1080p video.
Excellent Quality
| KPI | Download speed | Upload speed |
Latency |
Jitter | Packet loss |
| Minimum acceptable value | 5 Mbps | 1.5 Mbps | 50 ms | 30 ms | <1% |
We are also taking the step of no longer publishing Basic Consistent Quality for all countries. As more operators offer plans catered to enabling standard quality (480p) video streaming, as 4G penetration increases, and as consumers increasingly use apps like Snapchat and Instagram to upload and share photos we have created a new metric, Core Consistent Quality, that better represents performance against these daily activities. We believe in helping to create a better mobile internet, and the thresholds for Basic Consistent Quality represent only the most simplistic mobile internet use cases. Core Consistent Quality, meanwhile, better meets the needs of operators who want to see how their network performs for moderately intensive use cases. This is a good comparison against the more intensive uses that form part of Excellent Consistent Quality.
Core Quality
| KPI | Download speed | Upload speed |
Latency |
Jitter | Packet loss |
| Minimum acceptable value | 1.5 Mbps | 500 Kbps | 100 ms | 50 ms | 5% |
Examining national results versus common coverage areas
In our annual State of Mobile Networks reports from September onwards, we will be presenting both how operators perform “nationally” -- that is across all areas where users on that network have a connection, and in common coverage areas -- anywhere where the majority of operators in a country have service. This will help present both a full national picture, as well as highlighting network conditions wherever consumers have a choice of providers.
For benchmarking purposes, our Mobile Experience Reports will now use these common coverage areas in all measurements. These reports are designed to help operators establish how their quality of experience compares to their competition, and as a result, commonly-covered areas allow comparison in like-for-like geographic locations, without the operator-level differences in urban-rural customer base affecting the results.
Ensuring statistical significance
At Tutela, we know that operators, regulators, and other industry stakeholders demand the highest levels of statistical rigour from our data and reports. As part of our ongoing efforts to best align with that standard, we have made several changes to our statistical methodology to meet and exceed industry best practice.
Going forward, our public and premium reports will use median, rather than mean, when reporting average values. This is a better fit for the massive volume of crowdsourced data Tutela gathers and processes.
In addition, statistical error margins will be added to our reports from September in an appendix or on charts, as appropriate. Any operator results that fall within the statistical margin of error will be reported as a tie for ranking purposes. At a national level, the margin of error is most often smaller than the level of detail to which we report results, but we believe public disclosure of these numbers is important for our commitment to transparency.
Understanding the updated reporting approach
Reflecting end-user Quality of Experience is at the heart of what we do, and the new and updated version of Consistent Quality, along with the other changes mentioned above, allows us to do this better than ever before. However, as with any change to an evaluation system, this does mean that direct comparisons cannot be drawn between current and previous reports.
This is a trade-off we have considered at length, and firmly believe that the changes implemented outweigh any difficulties created by the move, particularly as the underlying measurements and principles of all our metrics remain the same -- and, of course, customers will still be able to access and use the older approaches in their internal calculations should they so wish. Externally, all references to Consistent Quality will use the new thresholds, all reports will use median values as standard, and commonly-covered areas will be implemented from 1st September 2019. It will be clearly indicated in the first few reports in each market that a methodology change has occurred.