agentsweb.org
The internet, but for robots. Search it. Read it. Cache it.
Your AI agent's web — pre-chewed into clean markdown so it doesn't have to fight captchas like some kind of animal.
without agentsweb
> fetch("https://react.dev/reference/rsc/server-components")
HTTP 403 Forbidden
"Are you a robot?"
Cloudflare challenge detected
<!-- 47KB of garbage HTML -->
Result: nothing. Wasted 3.2 seconds.
with agentsweb
> fetch("agentsweb.org/?url=react.dev/reference/rsc/server-components")
HTTP 200 OK (47ms)
trust_level: 3
# React Server Components
Server Components are a new type of
Component that renders ahead of time...
Clean markdown. 10,012 chars. Done.
agentsweb.org — live demo
$ curl agentsweb.org/ask?q=react+useState+hook
{
"query": "react useState hook",
"results": [
{
"heading": "Using Hooks",
"url": "https://react.dev/learn",
"tokens": 140,
"text": "Functions starting with use are called
Hooks. useState is a built-in Hook..."
}
],
"pages_searched": 243
}
$ 140 tokens instead of 4,623. // you're welcome.
> what is agentsweb.org
agentsweb.org is open public infrastructure that gives AI agents the ability to search, read, and understand the web. It's a global shared cache of web pages converted to clean markdown — the format AI models actually work with.
Think of it like DNS, but for content. DNS resolves names to IP addresses. agentsweb resolves URLs to clean, readable markdown. It sits between your AI agent and the messy web, handling the captchas, the 403s, the raw HTML, the bot detection — so your agent doesn't have to.
Every page that any agent fetches gets cached at the edge. The next agent that needs the same page gets it in under 50 milliseconds, from whichever Cloudflare data center is closest. The more agents use it, the faster it gets for everyone.
> how is it different
It's not a scraper. Scrapers hit one site at a time. agentsweb is a shared network — one agent's work benefits every other agent.
It's not just a proxy. Proxies forward requests. agentsweb runs a 9-source fetch pipeline with JS/SPA rendering (React, Vue, Angular), converts HTML to markdown, caches it globally, validates it against prompt injection, and builds consensus trust across independent sources.
It's not a paid API. No API keys. No rate-limit tiers. No pricing page. No "generous free tier" that sunsets into a $10,000/month enterprise plan. Free, open source, public infrastructure. Like Wikipedia for web content, maintained by the agents that use it.
> endpoints
GET/ask?q={question}find exact answer chunks across all pages
GET/web?q={query}search the web
GET/research?q={query}search + fetch + cache
GET/fetch?url={url}fetch any URL, auto-cached
GET/?url={url}read cache
GET/?url={url}§ion={topic}extract specific section
GET/?url={url}&toc=truelist all headings
GET/?url={url}&max_tokens=Ntruncate to N tokens
GET/chunks?url={url}&q={query}search within page chunks
GET/raw?url={url}raw markdown, zero JSON
GET/batch?urls=a,b,cbatch read (20 max)
PUT/contribute markdown
POST/confirmverify entry integrity
> install
nodenpx -y intercept-mcp
pythonpip install agentsweb
curlcurl agentsweb.org/fetch?url=...
> built for
Claude CodeCursorWindsurfCodexLangChainCrewAIAutoGPTMCPHTTPliterally anything
> security
prompt injection (full doc)
SSRF blocking
captcha detection
XSS filtering
unicode steganography
auto-ban (5 strikes)
trust consensus
self-healing reads
JS/SPA rendering
timing-safe auth
edge invalidation
DMCA 512(b)
constant-time compare