September 2020

Once a year, Tutela undertakes a comprehensive review of our public benchmarking methodology. We do this to ensure that what we measure — and how we measure it —  is as representative of the user experience of mobile subscribers as possible. Over time, user experience expectations and mobile application requirements evolve, while in parallel, Tutela makes enhancements to the metrics that we collect and analysis that we produce. With this in mind, our annual review aims to ensure we continue to report on the experience across each providers’ subscribers, wherever and whenever they are using their devices as fairly as possible while leveraging the latest technology developments available.  With these points in mind, we have made several updates to our standardized reporting methodology.

Adding coverage reporting

Tutela’s results have long represented the average experience of subscribers, wherever they happen to be using their devices. However, previously our focus has been solely on their experience of the network performance. From September 1st, we will also be evaluating operators based on their relative coverage, leveraging the development of our Network Coverage Product, released earlier this year.

Our reports will measure the area coverage provided by each operator, split into 4G/5G, 3G or 2G coverage across locations where any provider offers a mobile connection. 

Additions to Consistent Quality

Consistent Quality is Tutela’s metric that spans two tiers of mobile user experience, named “Excellent” and “Core”, that each represents a range of use cases. “Excellent” represents use cases including mobile HD video streaming, group video calls or streaming video games, while “Core” represents less demanding use cases like SD video streaming or web browsing. 

This year, we have added the KPI “Time to First Byte” to our Consistent Quality calculations to better represent the full experience of the user from the moment they attempt to connect to a service, rather than once the connection is established. This is because a slow initial connection time will make the download experience feel slower, even if the speed is fast once the connection is made. The time to first byte thresholds is set in line with the required download speed for each tier of Consistent Quality. For each tier, the maximum allowable Time to First Byte value is as follows:

• 3.2 seconds for Excellent Consistent Quality

• 10.67 seconds for Core Consistent Quality

We are also adding a measure of connection reliability to Consistent Quality, to reflect times when a connection is attempted but fails. In these cases, this will be considered a “fail” against both sets of thresholds. 

Measuring network experience by brands, not operators

In the past, we have included the performance of subscribers of flanker users into the results of the parent provider to encapsulate the full range of mobile subscriber experiences that an operator provides across all users of its network. From September 1st, we will now only report on the performance of the named brand – better reflecting the way consumers understand the brands, and the differing plan mixes that each operator providers under its different offerings. For example, our mobile user experience reporting will now consider AT&T and Cricket as separate providers.

Improvements to Common Coverage Areas

In 2019, Tutela introduced Common Coverage Areas to ensure the fairest comparison of subscriber experience between operators by removing edge-case locations where only a minority of operators provided coverage. Now that we are including a separate measure of network coverage, we are increasing the precision of these Common Coverage Areas, and limiting to only areas where all providers in a given comparison offer service. For example, if there are four national carriers presented in a report, records will only be taken from areas where customers of all of those four carriers have access to a mobile connection. This will include times when that connection is possible due to a domestic roaming agreement.

Summary of changes

At Tutela, we constantly strive to iterate and improve on our data collection and methodology to ensure that we’re publishing the most accurate, detailed, and representative data possible. The changes we’re making this year, which include the addition of geographic coverage scoring, improved reliability metrics, and the segmentation of flanker brands and MVNOs, should ensure that our reports are more useful than ever.