BOOMR.plugins. AutoXHR


Instrument and measure XMLHttpRequest (AJAX) requests.

With this plugin, sites can measure the performance of XMLHttpRequests (XHRs) and other in-page interactions after the page has been loaded.

This plugin also monitors DOM manipulations following a XHR to filter out "background" XHRs.

This plugin provides the backbone for the BOOMR.plugins.SPA plugin. Single Page App navigations use XHR and DOM monitoring to determine when the SPA navigations are complete.

This plugin has a corresponding Header Snippets that helps monitor XHRs prior to Boomerang loading.

For information on how to include this plugin, see the Building tutorial.

What is Measured

When AutoXHR is enabled, this plugin will monitor several events:

  • XMLHttpRequest requests
  • Fetch API requests
  • Clicks
  • window.History changes indirectly through SPA plugins (History, Angular, etc.)

When any of these events occur, AutoXHR will start monitoring the page for other events, DOM manipulations and other networking activity.

As long as the event isn't determined to be background activity (i.e an XHR that didn't change the DOM at all), the event will be measured until all networking activity has completed.

This means if your click generated an XHR that triggered an updated view to fetch more HTML that added images to the page, the entire event will be measured from the click to the last image completing.

Usage

To enable AutoXHR, you should set instrument_xhr to true:

BOOMR.init({
    instrument_xhr: true
});

Once enabled and initialized, the window.XMLHttpRequest object will be replaced with a "proxy" object that instruments all XHRs.

Monitoring XHRs

After AutoXHR is enabled, any XMLHttpRequest.send will be monitored:

xhr = new XMLHttpRequest();
xhr.open("GET", "/api/foo");
xhr.send(null);

If this XHR triggers DOM changes, a beacon will eventually be sent.

This beacon will have http.initiator=xhr and the beacon parameters will differ from a Page Load beacon. See BOOMR.plugins.RT and BOOMR.plugins.NavigationTiming for details.

Combining XHR Beacons

By default AutoXHR groups all XHR activity that happens in the same event together.

If you have one XHR that immediately triggers a second XHR, you will get a single XHR beacon. The u (URL) will be of the first XHR.

If you don't want this behavior, and want to measure every XHR on the page, you can enable alwaysSendXhr=true. When set, every distinct XHR will get its own XHR beacon.

BOOMR.init({
    AutoXHR: {
        alwaysSendXhr: true
    }
});

alwaysSendXhr can also be a list of strings (matching URLs), regular expressions (matching URLs), or a function which returns true for URLs to always send XHRs for.

BOOMR.init({
    AutoXHR: {
        alwaysSendXhr: [
            "http://domain.com/url/to/match",
            /regexmatch/,
        ]
   }
});

// or

BOOMR.init({
    AutoXHR: {
        alwaysSendXhr: function(url) {
            return url.indexOf("domain.com") !== -1;
        }
   }
});

Compatibility and Browser Support

Currently supported Browsers and platforms that AutoXHR will work on:

  • IE 9+ (not in quirks mode)
  • Chrome 38+
  • Firefox 25+

In general we support all browsers that support MutationObserver and XMLHttpRequest.

We will not use MutationObserver in IE 11 due to several browser bugs. See BOOMR.utils.isMutationObserverSupported for details.

Excluding Certain Requests or DOM Elements From Instrumentation

Whenever Boomerang intercepts an XMLHttpRequest (or Fetch), it will check if that request matches anything in the XHR exclude list. If it does, Boomerang will not instrument, time, send a beacon for that request, or include it in the BOOMR.plugins.SPA calculations.

There are two methods of excluding XHRs: Defining the BOOMR.xhr_excludes array, or using the excludeFilters option.

The BOOMR.xhr_excludes XHR excludes list is defined by creating a map, and adding URL parts that you would like to exclude from instrumentation. You can put any of the following in BOOMR.xhr_excludes:

  1. A full HREF
  2. A hostname
  3. A path

Example:

BOOMR = window.BOOMR || {};

BOOMR.xhr_excludes = {
  "www.mydomain.com":  true,
  "a.anotherdomain.net": true,
  "/api/v1/foobar":  true,
  "https://mydomain.com/dashboard/": true
};

The excludeFilters gives you more control by allowing you to specify one or more callbacks that will be run for each XHR/Fetch. If any callback returns true, the XHR/Fetch will not be instrumented.

BOOMR.init({
  AutoXHR: {
    excludeFilters: [
      function(anchor) {
        return anchor.href.match(/non-trackable/);
      }
    ]
  }
});

Finally, the domExcludeFilters DOM filters can be used to filter out specific DOM elements from being tracked (marked as "uninteresting").

BOOMR.init({
  AutoXHR: {
    domExcludeFilters: [
      function(elem) {
        return elem.id === "ignore";
      }
    ]
  }
});

Beacon Parameters

XHR beacons have different parameters in general than Page Load beacons.

  • Many of the timestamps will differ, see BOOMR.plugins.RT
  • All of the nt_* parameters are ResourceTiming, see BOOMR.plugins.NavigationTiming
  • u: the URL of the resource that was fetched
  • pgu: The URL of the page the resource was fetched on
  • http.initiator: xhr for both XHR and Fetch requests

Interesting Nodes

A MutationObserver is used to detect "interesting" nodes. Interesting nodes are new IMG/IMAGE/IFRAME/LINK (rel=stylesheet) nodes or existing nodes that are changing their source URL.

We consider the following "uninteresting" nodes:

  • Nodes that have either a width or height <= 1px.
  • Nodes with either a width or height of 0px.
  • Nodes that have display:none.
  • Nodes that have visibility:hidden.
  • Nodes with an opacity:0.
  • Nodes that update their source URL to the same value.
  • Nodes that have a blank source URL.
  • Images with a loading="lazy" attribute
  • Nodes that have a source URL starting with about:, javascript: or data:.
  • SCRIPT nodes because there is no consistent way to detect when they have loaded.
  • Existing IFRAME nodes that are changing their source URL because is no consistent way to detect when they have loaded.
  • Nodes that have a source URL that matches a AutoXHR exclude filter rule.
  • Nodes that have been manually excluded

Algorithm

Here's how the general AutoXHR algorithm works:

  • 0.0 SPA hard route change (page navigation)

    • Monitor for XHR resource requests and interesting Mutation resource requests for 1s or at least until page onload. We extend our 1s timeout after all interesting resource requests have completed.
  • 0.1 SPA soft route change from a synchronous call (eg. History changes as a result of a pushState or replaceState call)

    • In this case we get the new URL when the developer calls pushState or replaceState.
    • We create a pending event with the start time and the new URL.
    • We do not know if they plan to make an XHR call or use a dynamic script node, or do nothing interesting (eg: just make a div visible/invisible).
    • We also do not know if they will do this before or after they've called pushState/replaceState.
    • Our best bet is to monitor if either a XHR resource requests or interesting Mutation resource requests will happen in the next 1s.
    • When interesting resources are detected, we wait until they complete.
    • We restart our 1s timeout after all interesting resources have completed.
  • If something uninteresting happens, we set the timeout for 1 second if it wasn't already started.

    • We'll only do this once per event since we don't want to continuously extend the timeout with each uninteresting event.
  • If nothing happens during the additional timeout, we stop watching and fire the event. The event end time will be end time of the last tracked resource.

  • If nothing interesting was detected during the first timeout and the URL has not changed then we drop the event.

  • 0.2 SPA soft route change from an asynchronous call (eg. History changes as a result of the user hitting Back/Forward and we get a window.popstate event)

    • In this case we get the new URL from location.href when our event listener runs.
    • We do not know if this event change will result in some interesting network activity or not.
    • We do not know if the developer's event listener has already run before ours or if it will run in the future or even if they do have an event listener.
    • Our best bet is the same as 0.1 above.
  • 0.3 SPA soft route change from a click that triggers a XHR before the state is changed (when spaStartFromClick is enabled).

    • We store the time of the click
    • If any additional XHRs come next, we track those
    • When a pushState comes after the XHRs (before the timeout), we will "migrate" the click to a SPA event
  • 1 Click initiated (Only available when no SPA plugins are enabled)

    • User clicks on something.
    • We create a pending event with the start time and no URL.
    • We turn on DOM observer, and wait up to 50 milliseconds for activity.
      • If nothing happens during the first timeout, we stop watching and clear the event without firing it.
      • Else if something uninteresting happens, we set the timeout for 1s if it wasn't already started.
        • We'll only do this once per event since we don't want to continuously extend the timeout with each uninteresting event.
      • Else if an interesting node is added, we add load and error listeners and turn off the timeout but keep watching.
        • Once all listeners have fired, we start waiting again up to 50ms for activity.
      • If nothing happens during the additional timeout, we stop watching and fire the event.
  • 2 XHR/Fetch initiated

    • XHR or Fetch request is sent.
    • We create a pending event with the start time and the request URL.
    • We watch for all changes in XHR state (for async requests) and for load (for all requests).
    • We turn on DOM observer at XHR Onload or when the Fetch Promise resolves. We then wait up to 50 milliseconds for activity.
      • If nothing happens during the first timeout, we stop watching and clear the event without firing it.
      • If something uninteresting happens, we set the timeout for 1 second if it wasn't already started.
        • We'll only do this once per event since we don't want to continuously extend the timeout with each uninteresting event.
      • Else if an interesting node is added, we add load and error listeners and turn off the timeout.
        • Once all listeners have fired, we start waiting again up to 50ms for activity.
      • If nothing happens during the additional timeout, we stop watching and fire the event.

What about overlap?

  • 3.1 XHR/Fetch initiated while a click event is pending

    • If first click watcher has not detected anything interesting or does not have a URL, abort it
    • If the click watcher has detected something interesting and has a URL, then
      • Proceed with 2 above.
      • Concurrently, click stops watching for new resources
        • Once all resources click is waiting for have completed then fire the event.
  • 3.2 Click initiated while XHR/Fetch event is pending

    • Ignore click
  • 3.3 Click initiated while a click event is pending

    • If first click watcher has not detected anything interesting or does not have a URL, abort it.
    • Else proceed with parallel event steps from 3.1 above.
  • 3.4 XHR/Fetch initiated while an XHR/Fetch event is pending

    • Add the second XHR/Fetch as an interesting resource to be tracked by the XHR pending event in progress.
  • 3.5 XHR/Fetch initiated while SPA event is pending

    • Add the second XHR/Fetch as an interesting resource to be tracked by the XHR pending event in progress.
  • 3.6 SPA event initiated while an XHR event is pending

    • Proceed with 0 above.
    • Concurrently, XHR event stops watching for new resources. Once all resources the XHR event is waiting for have completed, fire the event.
  • 3.7 SPA event initiated while a SPA event is pending

    • If the pending SPA event had detected something interesting then send an aborted SPA beacon. If not, drop the pending event.
    • Proceed with 0 above.

Methods


addDomExcludeFilter(cb, ctx [, name])

Add a DOM filter function to the list of functions to run to validate if an DOM element should be instrumented.

Parameters:

Name Type Argument Description
cb BOOMR.plugins.AutoXHR.htmlElementCallback

Callback to run to validate filtering of an XHR Request

ctx Object

Context to run {@param cb} in

name string <optional>

Optional name for the filter, called out when running exclude filters for debugging purposes

Example

BOOMR.plugins.AutoXHR.addDomExcludeFilter(function(elem) {
  return elem.id === "ignore";
}, null, "exampleFilter");

addExcludeFilter(cb, ctx [, name])

Add a XHR filter function to the list of functions to run to validate if an XHR should be instrumented.

Parameters:

Name Type Argument Description
cb BOOMR.plugins.AutoXHR.htmlElementCallback

Callback to run to validate filtering of an XHR Request

ctx Object

Context to run {@param cb} in

name string <optional>

Optional name for the filter, called out when running exclude filters for debugging purposes

Example

BOOMR.plugins.AutoXHR.addExcludeFilter(function(anchor) {
  var m = anchor.href.match(/some-page\.html/g);

  // If matching flag to not instrument
  if (m && m.length > 0) {
    return true;
  }
  return false;
}, null, "exampleFilter");

enableAutoXhr()

Enables AutoXHR if not already enabled.


getMutationHandler()

Gets the MutationHandler

Returns:

Type: MutationHandler

Handler


getPathName(anchor)

Tries to resolve href links from relative URLs.

This implementation takes into account a bug in the way IE handles relative paths on anchors and resolves this by assigning a.href to itself which triggers the URL resolution in IE and will fix missing leading slashes if necessary.

Parameters:

Name Type Description
anchor string

The anchor object to resolve

Returns:

Type: string

The unrelativized URL href


init(config)

Initializes the plugin.

Parameters:

Name Type Description
config object

Configuration

Properties
Name Type Argument Default Description
instrument_xhr boolean <optional>

Whether or not to instrument XHR

AutoXHR.spaBackEndResources Array.<string> <optional>

Default resources to count as Back-End during a SPA nav

AutoXHR.monitorFetch boolean <optional>

Whether or not to instrument fetch()

AutoXHR.fetchBodyUsedWait number <optional>

If the fetch response's bodyUsed flag is false, we'll wait this amount of ms before checking RT for an entry. Setting to 0 will disable this feature

AutoXHR.alwaysSendXhr boolean | Array.<string> | Array.<RegExp> | Array.<function()> <optional>

Whether or not to send XHR beacons for every XHR.

captureXhrRequestResponse boolean <optional>

Whether or not to capture an XHR's request and response bodies on for the xhr_load event.

AutoXHR.spaIdleTimeout number <optional>
1000

Timeout for Single Page Applications after the final resource fetch has completed before calling the SPA navigation complete. Default is 1000ms.

AutoXHR.xhrIdleTimeout number <optional>
50

Timeout for XHRs after final resource fetch has completed before calling the XHR complete. Default is 50ms.

AutoXHR.xhrRequireChanges boolean <optional>
true

Whether or not a XHR beacon will only be triggered if there were DOM changes.

AutoXHR.spaStartFromClick boolean <optional>
false

In Single Page Apps, start tracking the SPA Soft Navigation from any preceeding clicks. If false, will start from the most recent pushState.

Returns:

BOOMR.plugins.AutoXHR The AutoXHR plugin for chaining


is_complete()

This plugin is always complete (ready to send a beacon)

Returns:

Type: boolean

true


loadFinished(resource)

Mark this as the time load ended via resources loadEventEnd property, if this resource has been added to the MutationHandler already notify that the resource has finished. Otherwise add this call to the lise of Events that occured.

Parameters:

Name Type Description
resource object

Resource


shouldExcludeXhr(anchor)

Based on the contents of BOOMR.xhr_excludes check if the URL that we instrumented as XHR request matches any of the URLs we are supposed to not send a beacon about.

Parameters:

Name Type Description
anchor HTMLAnchorElement

HTML anchor element with URL of the element checked against BOOMR.xhr_excludes

Returns:

Type: boolean

true if intended to be excluded, false if it is not in the list of excludables

Type Definitions


htmlElementCallback(elem)

A callback with a HTML element.

Parameters:

Name Type Description
elem HTMLAnchorElement

HTML element