82trust / 100
Site Crawl
by Site crawl (pages to markdown) in Search & web
x402 APIPassing, checked 3 h ago
Crawl a website from a starting URL and return each page as clean markdown: breadth-first over internal links, bounded by page count and depth, honouring robots.txt, with per-page title, status, depth and outbound links. Use it when an agent needs a whole section of a site rather than one known page.
POST https://agent402.tools/api/site-crawl
Last 30 days
All checks passedSome failedAll failedNot checked
- Uptime
- 100%
- Response time
- 191 ms typical, 210 ms slowest 5%
- Last check
- 3 h ago
- Next check
- any minute now
How to call it
# See the payment challenge (nothing is charged)
curl -i -X POST "https://agent402.tools/api/site-crawl" \
-H "content-type: application/json" \
-d '{"limit":3,"maxDepth":1,"url":"https://agent402.tools"}'import { wrapFetchWithPayment } from "@x402/fetch";
import { x402Client } from "@x402/core/client";
import { ExactEvmScheme } from "@x402/evm/exact/client";
import { privateKeyToAccount } from "viem/accounts";
const client = new x402Client().register(
"eip155:8453",
new ExactEvmScheme(privateKeyToAccount(process.env.AGENT_KEY)),
);
const pay = wrapFetchWithPayment(fetch, client);
// Not sure it's safe to pay? Preflight it first for $0.005:
// GET https://toolvet.app/api/v1/check?url=https%3A%2F%2Fagent402.tools%2Fapi%2Fsite-crawl
const res = await pay("https://agent402.tools/api/site-crawl", {
method: "POST",
headers: { "content-type": "application/json" },
body: JSON.stringify({"limit":3,"maxDepth":1,"url":"https://agent402.tools"}),
});
console.log(await res.json());Example input
{
"limit": 3,
"maxDepth": 1,
"url": "https://agent402.tools"
}Example output
{
"crawled": 3,
"elapsedMs": 645,
"fetchedAt": "2026-08-22T00:00:00.000Z",
"fetches": 4,
"format": "markdown",
"pages": [
{
"content": "# Agent402: 500+ pay-per-call tools for AI agents over x402 and MPP\n\nNo account. No API key. No card on file.\n…",
"contentChars": 8000,
"depth": 0,
"links": [
"https://agent402.tools/",
"https://agent402.tools/decide",
"https://agent402.tools/reports"
],
"status": 200,
"title": "Agent402: 500+ pay-per-call tools for AI agents over x402 and MPP",
"url": "https://agent402.tools/"
},
{
"content": "…",
"contentChars": 5104,
"depth": 1,
"links": [
"https://agent402.tools/"
],
"status": 200,
"title": "Agent402 Reports: research, 13F, insider flow, audits",
"url": "https://agent402.tools/reports"
},
{
"content": "…",
"contentChars": 5880,
"depth": 1,
"links": [
"https://agent402.tools/"
],
"status": 200,
"title": "Agent402 Decide: a plan for any job",
"url": "https://agent402.tools/decide"
}
],
"queued": 75,
"robotsTxt": "honoured",
"skipped": {
"binary": 0,
"depth": 22,
"error": 0,
"limit": 75,
"offsite": 4,
"pattern": 0,
"robots": 0,
"unsafe": 0
},
"source": "live fetch over internal links (breadth-first), robots.txt honoured for Agent402Bot",
"truncated": true,
"untrustedContent": true,
"url": "https://agent402.tools/"
}Security scan
- No findings. We scan names, descriptions and tool definitions for hidden instructions and other prompt-injection patterns.
Recent checks
| When | Result | HTTP | Time | Price |
|---|---|---|---|---|
| 3 h ago | Passed | 402 | 221 ms | $0.02 |
| 7 h ago | Passed | 402 | 195 ms | $0.02 |
| 11 h ago | Passed | 402 | 194 ms | $0.02 |
| 14 h ago | Passed | 402 | 185 ms | $0.02 |
| 19 h ago | Passed | 402 | 210 ms | $0.02 |
| 22 h ago | Passed | 402 | 191 ms | $0.02 |
| 25 h ago | Passed | 402 | 191 ms | $0.02 |
| 30 h ago | Passed | 402 | 181 ms | $0.02 |