improved
RUM update: bot list updates
January 20th, 2025
Our bot list is used to identify and exclude bot traffic from SpeedCurve RUM data. Today we have updated our bot list to add the following user-agents:
- AmazonProductDiscovery
- goodbot
- Any user-agent containing "/robot"
- Any user-agent containing "crawler"
The full bot list at the time of writing is below. All bots are identified by performing a substring match on the user-agent header, unless otherwise specified.
- /robot
- AdsBot-Google
- AmazonAdBot
- AmazonProductDiscovery
- APIs-Google
- Applebot
- Baiduspider
- bingbot
- BingPreview
- bot.html
- Bytespider
- Catchpoint
- Chrome-Lighthouse
- crawler
- Crawler
- Cypress
- facebookexternalhit
- FeedFetcher-Google
- Fundamental Monitor
- Ghost Inspector
- goodbot
- Google Page Speed Insights
- Google Web Preview
- Google-Read-Aloud
- Googlebot
- googleweblight
- HeadlessChrome
- https://deepcrawl.com/bot
- HubSpot Crawler
- JobBot
- Mediapartners-Google
- moatbot
- PetalBot
- PhantomJS
- Pingdom
- PTST
- quantcast
- Rigor
- RuxitSynthetic
- Webcrawler
- Yahoo Ad monitoring
- YandexBot
- YandexMobileBot
- Any user-agent identified as "bot" or "wget" by ua-parser