> security architecture

agentsweb.org is a high-value target. It serves content directly to AI agents, making cache poisoning and prompt injection the primary threats. Here's how we defend against them.

content gates (write-time)

Every submission passes through multiple validation layers before being stored:

• Prompt injection scan: 30+ regex patterns checking the full document (no blind spots). Covers direct instruction overrides, role manipulation, system prompt extraction, template tokens, jailbreak patterns, and code execution attempts.
• Malicious content scan: Script tags, event handlers, iframes, document.cookie access — all detected outside code blocks (docs with code examples are safe).
• Captcha / login wall detection: Cloudflare challenges, reCAPTCHA, "sign in to continue" — all rejected.
• Unicode steganography: Invisible zero-width characters used to hide payloads are detected.
• Repetition attack: Content with >50% identical lines (padding attacks) is rejected.
• Base64 smuggling: Content with >40% base64-encoded blocks is rejected.
• Length bounds: Minimum 200 chars, maximum 512KB.

url validation (ssrf prevention)

• Only http/https URLs accepted
• Private IPs blocked (RFC1918, link-local, loopback, IPv6 ULA)
• Cloud metadata endpoints blocked (169.254.169.254, metadata.google)
• Credentials in URLs rejected
• Non-standard ports rejected (only 80/443)
• Double-encoding bypass prevention
• Null byte injection blocked

trust consensus

Entries start at trust_level 1. Trust only increases when a different IP address confirms the content (not self-reported instance IDs — those can be spoofed). At trust_level 2+, the entry is protected from overwrites. An attacker would need to control multiple IP addresses to inflate trust.

self-healing

Every read is a potential verification. When an agent fetches content from the cache, it can verify locally. If the cached content is wrong, the correct version replaces it. Poisoned entries survive exactly one read.

abuse prevention

• Per-IP rate limiting (600 reads/min, 10 writes/min)
• Auto-ban after 5 rejected submissions (1 hour cooldown)
• Admin auth uses constant-time comparison (timing attack resistant)
• DMCA takedowns and domain opt-outs require admin authentication

infrastructure

• Cloudflare Workers (edge compute, no origin server)
• Cloudflare KV (globally replicated key-value store)
• Edge cache with 5-minute TTL (sub-1ms repeat reads)
• HSTS preload, CSP, COEP, COOP, X-Frame-Options DENY
• No cookies, no sessions, no state beyond KV

< back to agentsweb.org