A/B testing (RUM)

SpeedCurve is a SPA (single page app), so we construct the charts dynamically using JSONP. It works great, but we're always looking for ways to make the dashboards faster. 

One downside to making requests dynamically is that the browser preloader isn't used. This isn't a factor for later SPA requests, but on the first page view the preloader might still bring some benefits. 

Or maybe not. 

We weren't sure, so we ran an A/B test. Long story short: doing the first JSONP request via markup caused charts to render 300 milliseconds faster. 

How we ran the test

It's easy to run this A/B test using SpeedCurve RUM. 

1. The first step is to add a User Timing metric that marks when the JSONP response is received. 

We call this metric OnDataReceived. Some popular browsers don't support User Timing, so we use LUX's shim for mark() and measure() since it works in all browsers:

function OnDataReceived(jsondata) {
  LUX.measure("OnDataReceived");
  ...

The code above measures the time from NavigationStart to when the JSONP response is processed. (All User Timing marks and measures are automatically captured by LUX and displayed in customers' dashboards.)

2. The next step is to split users across two buckets.

In the control bucket, we continue to fetch the JSONP dynamically. In the test bucket we use markup to make the JSONP request. 

We use RUM's addData API to segment the RUM data across these buckets. The control bucket has JsonpMarkup=no, while the test bucket has JsonpMarkup=yes. 

Here's the backend PHP snippet:

if ( rand(0,1) ) {
  echo "<script>onGo();</script>\n"; // request JSONP dynamically
  echo "<script>LUX.addData('JsonpMarkup', 'no');</script>\n";
}
else {
  echo "<script src='/site-data?jsonp=OnDataReceived'></script>\n";
  echo "<script>LUX.addData('JsonpMarkup', 'yes');</script>\n";
}

3. Look at the results (in this case, within minutes).

Within a few minutes, the data shows that OnDataReceived is ~300 milliseconds faster for the JsonpMarkup=yes bucket. We let the A/B test run a few days to confirm that using markup for the initial JSONP request is faster thanks to the browser preloader.

Takeaways

If you make requests dynamically using JavaScript, you might be missing a performance opportunity by letting the preloader do its work. This isn't just true for JSONP requests, it also applies to images. 

If you're curious about this optimization, run an A/B test to see if making initial requests using markup makes your site faster and your users happier. We'd be happy to set you up with a free trial to make it easy peasy (lemon squeezy).