RUM update: bot list updates
5 months ago by Joseph
21 November 2023 (UTC)
Our bot list is used to identify and exclude bot traffic from SpeedCurve RUM data. Today we have updated our bot list to add the following user-agents:
- AmazonAdBot
- Bytespider
- Any user-agent containing "bot.html"
The full bot list at the time of writing is below. All bots are identified by performing a substring match on the user-agent header, unless otherwise specified.
- AdsBot-Google
- AmazonAdBot
- APIs-Google
- Applebot
- Baiduspider
- bingbot
- BingPreview
- bot.html
- Bytespider
- Catchpoint
- Chrome-Lighthouse
- Cypress
- FeedFetcher-Google
- Fundamental Monitor
- Ghost Inspector
- Google Page Speed Insights
- Google Web Preview
- Google-Read-Aloud
- Googlebot
- googleweblight
- HeadlessChrome
- https://deepcrawl.com/bot
- HubSpot Crawler
- JobBot
- Mediapartners-Google
- moatbot
- PetalBot
- PhantomJS
- Pingdom
- PTST
- quantcast
- RuxitSynthetic
- Webcrawler
- Yahoo Ad monitoring
- YandexBot
- YandexMobileBot
- Any user-agent identified as "wget" by ua-parser
- Any user-agent identified as "bot" by ua-parser