It's Not All About the Download Speeds
This is in regards to Techxtremist's Youtube video from the other day.
I used to always take Root Metrics scoring with a grain of salt but I'm seeing that the results are a lot more thorough and are consistent with everyone's usage habits. The problem with Root Metrics is that the results are most times out of date since they only release the results every quarter or so. Also, as was pointed out, there is value in doing download speed test using Ookla's app but this is not the whole picture. The NPerf app gives a better picture (in some cases) because it tests the load time of heavily used website (IE: Google, Youtube, amazon, etc) and also loads a sample Youtube video in different resolutions.
I also want to point out that the Ookla speed tests usually run from a datacenter that is is close to the user (even on the same carrier's network). In this regard, sometimes the results we see here are not indicative of real-world usage-patterns.
Think of this the same way we think of having fast load times/latency in regards to online gaming. In this case, a fast download is not all that is needed. Another example is when sending an Imessage. Sometimes when sending a quick text-based message, I may not see the it reach the network right away. Sometimes the issue could be the receiving Iphone, but there are times when the first part of the message getting sent (user device to home carrier) is slow. I see this type of issue more with T-Mobile than I do with AT&T and Verizon but haven't tried it as much recently.
The Nperf app and some of the measurements it does, got me thinking about other ways to measure the performance of the carriers. These tests would involve using the developer toolss that are installed with Chrome and using the hotspot on each of my phones with a wifi connection to my laptop.
I decided to test only two sites for this because Amazon and Youtube load a bunch of other content (videos, ads, etc) that make each load different (IE: we don't get the same content each time we load the page). This website (drivetester.us) and Wikipedia are more consistent each time I load the pages (no additional advertising/content mostly). I want to make sure each test I do is on similar ground.
To make sure this test is as clean as possible, I am doing the following after each carrier-test.
- Clearing the DNS cache on my computer.
- Disabling web page caching in the network tools menu.
- Ensuring that the wifi connection to my phone is strong (it is right next to my computer)
In my Chrome browser, I go to the "Settings" menu --> More Tools --> Developer Tools
This will bring up the "Developer Tools" menu. On this screen, I click on the "Network" tab, click "disable cache- no throttling" and hit the stop button on the recorder so we can start a new measurement. Only start the recording right before you run the test on a specific website (IE: Wikipedia, drivetester, etc) and then hit stop.
I also clear my DNS cache. On my Mac, I run the following command. You will be prompted for the administrator password:
sudo dscacheutil -flushcache; sudo killall -HUP mDNSResponder
You will only want to compare results each day/time you run the tests (IE: compare each carrier). For example, Google or drivetester.us may have other content on a different day so the load time will be different.
Verizon Tests
Loading drivetester.us
There's a lot going on here (IE: DNS request, loading of additional content, etc) so we will only look at the high-level info (IE: total load time)
The total load time for drivetester.us on Verizon hotspot was 3.09 second. There were 18 requests total, 352kb transferred. We want to ensure the follow-on tests (other carriers) match in regards to requests and transferred content (otherwise the content is not static enough).
Loading Wikipedia
Total load time is 645 milleseconds.
AT&T Test
Loading drivetester.us
1.28 seconds load time
Loading Wikipedia
Total load time 1.36 seconds
T-Mobile Test
Loading drivetester.us
1.96 seconds load time
Loading Wikipedia
589 milleseconds load time
This was by no means a thorough test. It would be cumbersome to do this all the time. Having a mobile app like Nperf makes this process easier.
I am just using this article to point out that download speed is not all that is important. Ping time, web page loads, other latency, jitter can cause problems that we see more with everyday use-patterns.