improved
RUM update: bot list updates
about 4 hours ago by Joseph
21 January 2025 (UTC)
Our bot list is used to identify and exclude bot traffic from SpeedCurve RUM data. Today we have updated our bot list to add the following user-agents:
- AmazonProductDiscovery
- goodbot
- Any user-agent containing "/robot"
- Any user-agent containing "crawler"
The full bot list at the time of writing is below. All bots are identified by performing a substring match on the user-agent header, unless otherwise specified.
- /robot
- AdsBot-Google
- AmazonAdBot
- AmazonProductDiscovery
- APIs-Google
- Applebot
- Baiduspider
- bingbot
- BingPreview
- bot.html
- Bytespider
- Catchpoint
- Chrome-Lighthouse
- crawler
- Crawler
- Cypress
- facebookexternalhit
- FeedFetcher-Google
- Fundamental Monitor
- Ghost Inspector
- goodbot
- Google Page Speed Insights
- Google Web Preview
- Google-Read-Aloud
- Googlebot
- googleweblight
- HeadlessChrome
- https://deepcrawl.com/bot
- HubSpot Crawler
- JobBot
- Mediapartners-Google
- moatbot
- PetalBot
- PhantomJS
- Pingdom
- PTST
- quantcast
- Rigor
- RuxitSynthetic
- Webcrawler
- Yahoo Ad monitoring
- YandexBot
- YandexMobileBot
- Any user-agent identified as "bot" or "wget" by ua-parser