Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions bot_categories.yml
Original file line number Diff line number Diff line change
Expand Up @@ -110,6 +110,7 @@ ai_agent:
- aranet-searchbot
- cartai agentic commerce
- chatgpt agent
- claude-agent
- claude-code
- claude-web
- crawl4ai
Expand All @@ -119,6 +120,7 @@ ai_agent:
- iaskspider
- kimi-agent
- kunatocrawler
- minimaxagent
- novaact
- opencode
- payhawk invoice fetching agent
Expand All @@ -136,11 +138,14 @@ ai_assistant:
- exauser
- geisthaus-pagefetcher
- gemini-deep-research
- google-gemininotebook
- googleagent-urlcontext
- haiku-searchbot
- henkbot
- kagi-fetcher
- kimi-user
- klaviyoaibot
- liner-user
- linerbot
- meta-externalfetcher
- mistralai-user
Expand Down Expand Up @@ -587,6 +592,7 @@ monitoring_and_analytics:
- rakuten image extraction bot
- rigor)
- ruxitsynthetic
- scooterza
- scouturlmonitor
- sentinel-linkcheck
- sentry/
Expand Down
23 changes: 23 additions & 0 deletions bot_ip_ranges.yml
Original file line number Diff line number Diff line change
Expand Up @@ -975,6 +975,16 @@ google:
- 136.124.136.0/27
- 136.124.136.32/27
- 136.124.136.64/27
- 136.124.142.0/27
- 136.124.142.32/27
- 136.124.142.64/27
- 136.124.142.96/27
- 136.124.142.128/27
- 136.124.143.0/27
- 136.124.143.32/27
- 136.124.144.0/27
- 136.124.144.32/27
- 136.124.144.64/27
- 142.250.32.0/27
- 142.250.32.32/27
- 142.250.32.64/27
Expand Down Expand Up @@ -2178,9 +2188,19 @@ google:
- 2600:1900:0:3d5::/64
- 2600:1900:0:3d6::/64
- 2600:1900:0:3d7::/64
- 2600:1900:0:3d8::/64
- 2600:1900:0:3d9::/64
- 2600:1900:0:3da::/64
- 2600:1900:0:3db::/64
- 2600:1900:0:3dc::/64
- 2600:1900:0:3f0::/64
- 2600:1900:0:3f1::/64
- 2600:1900:0:3f2::/64
- 2600:1900:0:410::/64
- 2600:1900:0:411::/64
- 2600:1900:0:412::/64
- 2600:1900:0:430::/64
- 2600:1900:0:431::/64
bing:
- 13.66.139.0/24
- 13.66.144.0/24
Expand All @@ -2202,6 +2222,7 @@ bing:
- 40.79.131.208/28
- 40.79.186.176/28
- 51.105.67.0/28
- 52.130.224.112/28
- 52.167.144.0/24
- 52.231.148.0/28
- 65.55.210.0/24
Expand Down Expand Up @@ -2344,6 +2365,8 @@ openai:
- 51.8.155.48/28
- 51.8.155.112/28
- 51.57.0.96/28
- 51.57.158.96/28
- 51.57.158.112/28
- 51.59.24.64/28
- 51.59.24.80/28
- 51.59.48.80/28
Expand Down
6 changes: 6 additions & 0 deletions bots.yml
Original file line number Diff line number Diff line change
Expand Up @@ -525,6 +525,7 @@ citeseerxbot: citeseerxbot
citoid: Citoid
claritybot/: ClarityBot
classla-web: CLASSLA-web
claude-agent: Claude-Agent
claude-code: Claude-Code
claude-searchbot: Claude-SearchBot
claude-user: Claude-User
Expand Down Expand Up @@ -778,6 +779,7 @@ google-apps-script: Google-Apps-Script
google-area120: Google-Area120
google-certificates-bridge: Google-Certificates-Bridge
google-firebase: Google-Firebase
google-gemininotebook: Google-GeminiNotebook
google-inspectiontool: Google-InspectionTool
google-physicalweb: Google-PhysicalWeb
google-read-aloud: Google-Read-Aloud
Expand Down Expand Up @@ -810,6 +812,7 @@ gslfbot: gslfbot
gtmetrix: GTmetrix
guestpostsbot/: GuestpostsBot
gulper web bot: Gulper Web Bot
haiku-searchbot: Haiku-SearchBot
halobot: HaloBot
hanaleibot: HanaleiBot
happywing: HappyWing
Expand Down Expand Up @@ -945,6 +948,7 @@ let's encrypt: Let's Encrypt
level9searchbot/: Level9SearchBot
library of congress web archiving: Library Of Congress Web Archiving
lightspeedsystemscrawler: LightspeedSystemsCrawler
liner-user: Liner-User
linerbot: LinerBot
linespider: Linespider
linguee bot: Linguee Bot
Expand Down Expand Up @@ -1014,6 +1018,7 @@ mindupbot: mindUpBot
miniature.io/: Miniature.io
minicrawler: minicrawler
miniflux/: Miniflux
minimaxagent: MiniMaxAgent
minoru's fediverse crawler: Minoru's Fediverse Crawler
mirrorweb.com: mirrorweb.com
mirrorwebcrawler: MirrorWebCrawler
Expand Down Expand Up @@ -1260,6 +1265,7 @@ sbintuitionsbot: SBIntuitionsBot
sbl-bot: SBL-BOT
schema-markup-validator: Schema-Markup-Validator
scoop.it/: Scoop.it
scooterza: ScooterZA
scope3: Scope3
scoutjet: ScoutJet
scrapeherobot/: ScrapeheroBot
Expand Down
2 changes: 1 addition & 1 deletion data/bot_categories.json

Large diffs are not rendered by default.

2 changes: 1 addition & 1 deletion data/bot_ip_ranges.json

Large diffs are not rendered by default.

2 changes: 1 addition & 1 deletion data/bots.json

Large diffs are not rendered by default.

40 changes: 20 additions & 20 deletions docs/bot_coverage.json
Original file line number Diff line number Diff line change
Expand Up @@ -21,24 +21,24 @@
},
{
"category": "ai_agent",
"user_agents": 35,
"user_agents": 37,
"upstream_bots": 12,
"agentry_bots": 21,
"agentry_categorized": 28
"agentry_bots": 23,
"agentry_categorized": 30
},
{
"category": "ai_assistant",
"user_agents": 35,
"upstream_bots": 19,
"agentry_bots": 34,
"agentry_categorized": 34
"user_agents": 38,
"upstream_bots": 20,
"agentry_bots": 37,
"agentry_categorized": 37
},
{
"category": "ai_crawler",
"user_agents": 186,
"upstream_bots": 150,
"agentry_bots": 183,
"agentry_categorized": 183
"user_agents": 187,
"upstream_bots": 151,
"agentry_bots": 184,
"agentry_categorized": 184
},
{
"category": "ai_search",
Expand Down Expand Up @@ -77,10 +77,10 @@
},
{
"category": "monitoring_and_analytics",
"user_agents": 260,
"user_agents": 261,
"upstream_bots": 141,
"agentry_bots": 258,
"agentry_categorized": 256
"agentry_bots": 259,
"agentry_categorized": 257
},
{
"category": "page_preview",
Expand Down Expand Up @@ -112,19 +112,19 @@
}
],
"total": {
"user_agents": 2326,
"upstream_bots": 1641,
"agentry_bots": 2264,
"agentry_categorized": 2270
"user_agents": 2333,
"upstream_bots": 1643,
"agentry_bots": 2271,
"agentry_categorized": 2277
},
"recent_crawlers": {
"added_since": "2024-01-01",
"left_out_tags": [
"browser-automation",
"scanner"
],
"user_agents": 882,
"user_agents": 883,
"upstream_bots": 529,
"agentry_bots": 871
"agentry_bots": 872
}
}
14 changes: 7 additions & 7 deletions docs/bots.html
Original file line number Diff line number Diff line change
Expand Up @@ -49,7 +49,7 @@ <h1 class="page-title">Bots</h1>
<h2 id="lists-title">The lists</h2>
<!-- render:bot-lists -->
<div class="tiles">
<div class="tile"><span class="tile-value">1,566</span><span class="tile-label">named bots</span><span class="tile-detail"><code>bots.yml</code>, keyed by a lowercase substring of the user agent</span></div>
<div class="tile"><span class="tile-value">1,572</span><span class="tile-label">named bots</span><span class="tile-detail"><code>bots.yml</code>, keyed by a lowercase substring of the user agent</span></div>
<div class="tile"><span class="tile-value">16</span><span class="tile-label">categories</span><span class="tile-detail"><code>bot_categories.yml</code>: <code>ai_crawler</code>, <code>search_engine_crawler</code>, <code>scanner</code>…</span></div>
<div class="tile"><span class="tile-value">6</span><span class="tile-label">search engines</span><span class="tile-detail"><code>search_engines.yml</code>: answer <code>bot.search_engine?</code></span></div>
<div class="tile"><span class="tile-value">4</span><span class="tile-label">exceptions</span><span class="tile-detail"><code>bot_exceptions.yml</code>: never a bot, whatever a matcher says</span></div>
Expand All @@ -67,27 +67,27 @@ <h2 id="coverage-title">Coverage</h2>
<p>How many of the bots of crawler-user-agents and ai.robots.txt each gem calls a bot, per category: every user agent crawler-user-agents gives as an instance, and, for each name of ai.robots.txt, which gives none, the user agent an AI agent of 2026 sends. <code>script/bot_coverage.rb</code> counts them into <code>docs/bot_coverage.json</code> and the weekly sync regenerates it, with this table.</p>
</div>
<!-- render:coverage -->
<p class="note">Of the 882 crawler user agents added to crawler-user-agents since 2024 (test drivers and scanners left out), browser misses 353 and Agentry 11. Against fnando/browser dd88933.</p>
<p class="note">Of the 883 crawler user agents added to crawler-user-agents since 2024 (test drivers and scanners left out), browser misses 354 and Agentry 11. Against fnando/browser dd88933.</p>
<div class="table-scroll"><table class="options">
<thead><tr><th scope="col">Category</th><th scope="col" class="number">User agents</th><th scope="col" class="number">browser: <code>bot?</code></th><th scope="col" class="number">Agentry: <code>bot?</code></th><th scope="col" class="number">Agentry: a category</th></tr></thead>
<tbody>
<tr><td><code>academic_research</code></td><td class="number">35</td><td class="number">24</td><td class="number">35</td><td class="number">34</td></tr>
<tr><td><code>advertising_and_marketing</code></td><td class="number">97</td><td class="number">78</td><td class="number">94</td><td class="number">94</td></tr>
<tr><td><code>ai_agent</code></td><td class="number">35</td><td class="number">12</td><td class="number">21</td><td class="number">28</td></tr>
<tr><td><code>ai_assistant</code></td><td class="number">35</td><td class="number">19</td><td class="number">34</td><td class="number">34</td></tr>
<tr><td><code>ai_crawler</code></td><td class="number">186</td><td class="number">150</td><td class="number">183</td><td class="number">183</td></tr>
<tr><td><code>ai_agent</code></td><td class="number">37</td><td class="number">12</td><td class="number">23</td><td class="number">30</td></tr>
<tr><td><code>ai_assistant</code></td><td class="number">38</td><td class="number">20</td><td class="number">37</td><td class="number">37</td></tr>
<tr><td><code>ai_crawler</code></td><td class="number">187</td><td class="number">151</td><td class="number">184</td><td class="number">184</td></tr>
<tr><td><code>ai_search</code></td><td class="number">45</td><td class="number">36</td><td class="number">44</td><td class="number">43</td></tr>
<tr><td><code>archiver</code></td><td class="number">68</td><td class="number">50</td><td class="number">68</td><td class="number">68</td></tr>
<tr><td><code>browser_automation</code></td><td class="number">23</td><td class="number">2</td><td class="number">2</td><td class="number">21</td></tr>
<tr><td><code>feed_fetcher</code></td><td class="number">92</td><td class="number">40</td><td class="number">91</td><td class="number">91</td></tr>
<tr><td><code>http_library</code></td><td class="number">103</td><td class="number">68</td><td class="number">102</td><td class="number">102</td></tr>
<tr><td><code>monitoring_and_analytics</code></td><td class="number">260</td><td class="number">141</td><td class="number">258</td><td class="number">256</td></tr>
<tr><td><code>monitoring_and_analytics</code></td><td class="number">261</td><td class="number">141</td><td class="number">259</td><td class="number">257</td></tr>
<tr><td><code>page_preview</code></td><td class="number">136</td><td class="number">110</td><td class="number">135</td><td class="number">133</td></tr>
<tr><td><code>scanner</code></td><td class="number">108</td><td class="number">46</td><td class="number">106</td><td class="number">106</td></tr>
<tr><td><code>search_engine_crawler</code></td><td class="number">418</td><td class="number">384</td><td class="number">417</td><td class="number">406</td></tr>
<tr><td><code>search_engine_optimization</code></td><td class="number">685</td><td class="number">481</td><td class="number">674</td><td class="number">671</td></tr>
</tbody>
<tfoot><tr><th scope="row">Total</th><td class="number">2,326</td><td class="number">1,641</td><td class="number">2,264</td><td class="number">2,270</td></tr></tfoot>
<tfoot><tr><th scope="row">Total</th><td class="number">2,333</td><td class="number">1,643</td><td class="number">2,271</td><td class="number">2,277</td></tr></tfoot>
</table></div>
<!-- /render:coverage -->
<div class="prose">
Expand Down
4 changes: 2 additions & 2 deletions docs/index.html
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,7 @@ <h1 class="wordmark">Agentry</h1>
<div>
<p class="summary">Agentry is a Ruby gem that reads the <code>User-Agent</code> header, the line of text every browser and crawler sends with a request, and tells you the browser, the operating system, the device or the bot behind it. It is a fork of the browser gem: the same API, faster, with a bot list updated every week.</p>
<!-- render:lede -->
<p class="summary">25 µs per request instead of 126 µs with browser, 2.0 µs once a user agent was seen (YJIT). Of the 882 crawler user agents added to crawler-user-agents since 2024 (test drivers and scanners left out), browser misses 353 and Agentry 11.</p>
<p class="summary">25 µs per request instead of 126 µs with browser, 2.0 µs once a user agent was seen (YJIT). Of the 883 crawler user agents added to crawler-user-agents since 2024 (test drivers and scanners left out), browser misses 354 and Agentry 11.</p>
<!-- /render:lede -->
<code class="install">gem "agentry"</code>
<p class="links"><a href="guide.html">Guide</a><a href="guide.html#migrating">Coming from browser</a><a href="bots.html">Bots</a></p>
Expand Down Expand Up @@ -79,7 +79,7 @@ <h2 id="numbers-title">In numbers</h2>
<div class="tiles">
<div class="tile"><span class="tile-value">25 µs</span><span class="tile-label">per request</span><span class="tile-detail">instead of 126 µs with browser 6.2.0, on 2,000 distinct user agents</span></div>
<div class="tile"><span class="tile-value">2.0 µs</span><span class="tile-label">for a repeated user agent</span><span class="tile-detail">instead of 114 µs: detection is cached per user agent</span></div>
<div class="tile"><span class="tile-value">353</span><span class="tile-label">recent crawlers browser misses</span><span class="tile-detail">of the 882 crawler user agents added to crawler-user-agents since 2024 (test drivers and scanners left out); Agentry misses 11</span></div>
<div class="tile"><span class="tile-value">354</span><span class="tile-label">recent crawlers browser misses</span><span class="tile-detail">of the 883 crawler user agents added to crawler-user-agents since 2024 (test drivers and scanners left out); Agentry misses 11</span></div>
</div>
<p class="note more">Ruby 4.0.9, Apple M4 Max, measured on 2026-11-04: browser 6.2.0 at <code>dd88933</code>, Agentry 1.0.0 at <code>e8bdc86</code>. Each figure is the median of 7 runs. YJIT on.</p>
<!-- /render:numbers -->
Expand Down
Loading