CLS scores in RUM vs synthetic

While the method for calculating CLS is consistent across tools, there are a couple of gotchas to be aware of when comparing mixed data sets (e.g., real user data versus synthetic test data).

1. Differences in how the page's life cycle is measured

CLS is an accumulation of layout shifts that occur during a page's life cycle. However, measurement of the life cycle may differ across technologies.

• When measuring with RUM, most tools measure the accumulation of layout shifts that happen before the load event, but stop there.
• The CrUX data set (an invaluable resource provided by Google) measures CLS for Chrome users*, but extends the life cycle until the viewability state of the browser window has changed.
• With synthetic monitoring tools, by default you'll continue to capture layout shifts until the page is considered 'fully loaded'. Most likely, 'fully loaded' happens well after the load event is fired, but as there is no visibility state change due to the absence of a user interaction, the life cycle duration will likely be different.

The differences in the duration of the page life cycle can account for some of the larger discrepancies we've seen in CLS.

2. Breadth and variety of user environments

Synthetic monitoring is well known for creating a "lab" environment, where there are few, if any, changes between measurements. Browser version, device, viewport size, network throttling, and CPU throttling all remain constant. This allows you to more accurately baseline the target application.

In the real world, however, you have a complex distribution of various devices and environmental conditions. This can lead to a bit of variability when accumulating layout shifts. If a viewport is much smaller, some shifts may occur below the fold. As network conditions vary, you may see differences in CLS due to slow/fast loading of fonts, and so on.

The point is, measurements in the wild will vary when comparing to a baseline synthetic measurement (or perhaps a Lighthouse test run).

*CrUX captures data from Chrome for users who have opted-in, have not set up a sync passphrase, and have usage statistic reporting enabled.