Consistent Quality

For as long as consumers have been downloading apps, conventional wisdom has suggested that user satisfaction and download speed go hand in hand. The assumption made sense: home internet speeds are packaged and advertised by the megabit; the data speed increase has been the most notable user change in going from 2G to 5G; and, not insignificantly, mobile network providers have been running adverts based on speed for years.

But as networks have evolved, the average download speed has crept ever higher, while subscribers haven’t necessarily been getting happier with the quality of their mobile experience.

Why speed doesn't tell the whole story

In a study by the Wall Street Journal, 53 journalists assessed their internet usage over a period of months by streaming popular services such as Netflix, Amazon Prime, and YouTube simultaneously. What they found was that "quality didn't improve much with higher speeds. Picture clarity was about the same. Videos didn't launch quicker." Fundamentally, the subjects of the study just weren't using most of their available bandwidth. 

Similarly, a 2015 study from the Boston Consulting Group tested key network performance indicators against a survey of users’ happiness with their connection and found that even when streaming video, any increase in speed over just 1.5 Mbps saw little increase in user satisfaction.

Looking at the minimum requirements from streaming video providers, you can see why. Popular services like YouTube, Netflix, and Hulu — which account for a significant proportion of mobile data traffic — have a minimum requirement of 1 Mbps to stream SD video, while even 720p or 1080p streams only require a maximum of 4 or 5 Mbps.

Introducing Consistent Quality 

So if download speeds aren’t a satisfactory proxy for a ‘good’ network experience, what is? Tutela has designed a set of metrics, named Consistent Quality, which use the widely-accepted network requirements of different use cases to categorize the quality of connections.

We measure six key performance indicators — the average download speed, upload speed, latency, jitter, packet loss, and time to first byte — and calculate what percentage of tests meet or exceed our two thresholds, Core and Excellent Consistent Quality. 

In addition, we have added “reliability” into Consistent Quality – that is, the percentage of tests attempted which did not succeed due to a connectivity issue on either the download or server response component. These will contribute to the total percentage of Consistent Quality tests that are considered to have failed both the Core and the Excellent Consistent Quality thresholds.

The Excellent Consistent Quality thresholds have been designed using the latest available network performance recommendations from services like Netflix, Skype (including Skype for Business) and others, as well as emerging use cases such as live video streaming on services like YouTube and streaming video games.

Excellent Consistent Quality

Core Consistent Quality is designed to represent common but less-demanding use-cases than Excellent Consistent Quality, specifically standard-definition video streaming, web browsing, email use, and photo-sharing apps like Snapchat and Instagram. 

Core Consistent Quality

Understanding the Key Performance Indicators 

Including download and upload speeds in any test remains important: even if it’s not the only thing that matters, a 128 kbps connection still won’t work for any of our most important apps. A download speed of 5 Mbps is sufficient to stream HD video from any of the most popular streaming services, while the Core Consistent Quality threshold of 1.5 Mbps is in line with streaming standard-definition video — and, in many cases, is the maximum speed that some carriers will allow standard-definition video to be streamed at. 

Users download significantly more data than they upload, making a faster download speed the more important of the two, but our Excellent Consistent Quality upload speed threshold of 1.5 Mbps means users can conduct a video chat, upload images or videos to Snapchat, or send emails with large attachments without noticeable delay. 

Once the throughput thresholds have been reached, latency is arguably the biggest difference-maker in user experience. Our threshold of 50 ms is fast enough for online games like Fortnite, as well as voice and video calling. At faster download speeds, lower latency is also the most significant factor in web page load time. Using the 100 ms latency threshold for our Core quality measurement ensures a connection is still usable, although subscribers will notice a difference in performance. 

Jitter is a measure of the variation in latency, and has a major impact on the quality of real-time services like voice and video calling. A low average latency is fine, but if the packet latency jumps between 25 ms and 75 ms regularly, users will see video artefacts and jumpiness, which degrades the experience. The excellent quality jitter threshold of less than 12 ms ensures that operators are achieving a consistently low latency, rather than just a low average.