A Comparative Analysis of Ookla's Speedtest and Measurement Lab's Network Diagnostic Test

Consumers, regulators, and internet service providers (ISPs) all use client-based "speed tests" to measure network performance, both in single-user settings and in aggregate. Two prevalent speed tests, Ookla's Speedtest and Measurement Lab's Network Diagnostic Test (NDT), are often used for similar purposes, despite having significant differences in both the test design and implementation and in the infrastructure used to conduct measurements. This paper presents a comparative evaluation of Ookla and NDT7 (the latest version of NDT), both in controlled and wide-area settings. The results show that Ookla and NDT7 report similar speeds when the latency between the client and server is low, but that the tools diverge when path latency is high. To characterize the behavior of these tools in wide-area deployment, the researchers collected more than 40,000 pairs of Ookla and NDT7 measurements across six months and 67 households, with a range of ISPs and speed tiers. The analysis demonstrates various systemic issues, including high variability in NDT7 test results and systematically under-performing servers in the Ookla network.