On 1 July 2020, we updated our test agents. Our new test agents include updates to several pieces of core software, including:

  • Chrome has been updated from version 78.0 to 83.0.
  • Firefox has been updated from version 70.0 to 77.0.
  • Lighthouse has been updated from version 5.6.0 to 6.1.0.
  • Lighthouse test runs now use the official throttling settings for mobile and desktop.
  • Incremental updates to wptagent.

We have been testing these new agents in parallel with our existing agents. While we can't predict exactly how the updates will affect your metrics, here are a few observations we have made based on the sites we've looked at:

  • Lighthouse performance scores for mobile tests may stay roughly the same.
  • Lighthouse performance scores for desktop tests could be higher. This is due to changes in throttling. SpeedCurve used the default mobile throttling for all tests in Lighthouse v5.6. Desktop tests in Lighthouse v6.1 now use a much faster network connection.
  • Visual metrics like Start Render and Hero Element Render stay roughly the same. Improvements have been made to how Largest Contentful Paint is calculated that may change its value for some pages.
  • Total JS CPU time is reduced in almost all tests, with other CPU-related metrics like First CPU Idle being slightly reduced as well.

Some of our customers are seeing significant changes in some metrics – particularly JavaScript/CPU metrics, and especially on mobile. Here are some reasons why CPU-based metrics – such as Time to Interactive and Long Tasks – might be affected:

  • Chrome 79 (V8 7.9) had some performance improvements that would have been noticeable on most pages.
  • Chrome 80 (V8 8.0) also had some improvements that would affect pages with heavy memory usage.
  • We switched to a newer generation of EC2 instance that might improve performance on CPU-heavy pages.

Recommendations

  1. Add a note to your charts that highlights the test agent changes on 1 July 2020. That way other people on your teams will understand what caused potential changes.
  2. Update your performance budgets, if necessary, to reflect changes in your metrics.
  3. Remember that the absolute numbers in your charts don't matter as much as the relative changes you see in your numbers. After you've completed recommendations 1 and 2, focus on the improvement you see as you improve your code base, rather than on absolute numbers.

Did this answer your question?