82trust / 100

Site Crawl

by Site crawl (pages to markdown) in Search & web

x402 APIPassing, checked 3 h ago

Crawl a website from a starting URL and return each page as clean markdown: breadth-first over internal links, bounded by page count and depth, honouring robots.txt, with per-page title, status, depth and outbound links. Use it when an agent needs a whole section of a site rather than one known page.

POST https://agent402.tools/api/site-crawl

Last 30 days

All checks passedSome failedAll failedNot checked
Uptime
100%
Response time
191 ms typical, 210 ms slowest 5%
Last check
3 h ago
Next check
any minute now

How to call it

# See the payment challenge (nothing is charged)
curl -i -X POST "https://agent402.tools/api/site-crawl" \
  -H "content-type: application/json" \
  -d '{"limit":3,"maxDepth":1,"url":"https://agent402.tools"}'
import { wrapFetchWithPayment } from "@x402/fetch";
import { x402Client } from "@x402/core/client";
import { ExactEvmScheme } from "@x402/evm/exact/client";
import { privateKeyToAccount } from "viem/accounts";

const client = new x402Client().register(
  "eip155:8453",
  new ExactEvmScheme(privateKeyToAccount(process.env.AGENT_KEY)),
);
const pay = wrapFetchWithPayment(fetch, client);

// Not sure it's safe to pay? Preflight it first for $0.005:
// GET https://toolvet.app/api/v1/check?url=https%3A%2F%2Fagent402.tools%2Fapi%2Fsite-crawl
const res = await pay("https://agent402.tools/api/site-crawl", {
  method: "POST",
  headers: { "content-type": "application/json" },
  body: JSON.stringify({"limit":3,"maxDepth":1,"url":"https://agent402.tools"}),
});
console.log(await res.json());

Example input

{
  "limit": 3,
  "maxDepth": 1,
  "url": "https://agent402.tools"
}

Example output

{
  "crawled": 3,
  "elapsedMs": 645,
  "fetchedAt": "2026-08-22T00:00:00.000Z",
  "fetches": 4,
  "format": "markdown",
  "pages": [
    {
      "content": "# Agent402: 500+ pay-per-call tools for AI agents over x402 and MPP\n\nNo account. No API key. No card on file.\n…",
      "contentChars": 8000,
      "depth": 0,
      "links": [
        "https://agent402.tools/",
        "https://agent402.tools/decide",
        "https://agent402.tools/reports"
      ],
      "status": 200,
      "title": "Agent402: 500+ pay-per-call tools for AI agents over x402 and MPP",
      "url": "https://agent402.tools/"
    },
    {
      "content": "…",
      "contentChars": 5104,
      "depth": 1,
      "links": [
        "https://agent402.tools/"
      ],
      "status": 200,
      "title": "Agent402 Reports: research, 13F, insider flow, audits",
      "url": "https://agent402.tools/reports"
    },
    {
      "content": "…",
      "contentChars": 5880,
      "depth": 1,
      "links": [
        "https://agent402.tools/"
      ],
      "status": 200,
      "title": "Agent402 Decide: a plan for any job",
      "url": "https://agent402.tools/decide"
    }
  ],
  "queued": 75,
  "robotsTxt": "honoured",
  "skipped": {
    "binary": 0,
    "depth": 22,
    "error": 0,
    "limit": 75,
    "offsite": 4,
    "pattern": 0,
    "robots": 0,
    "unsafe": 0
  },
  "source": "live fetch over internal links (breadth-first), robots.txt honoured for Agent402Bot",
  "truncated": true,
  "untrustedContent": true,
  "url": "https://agent402.tools/"
}

Security scan

  • No findings. We scan names, descriptions and tool definitions for hidden instructions and other prompt-injection patterns.

Recent checks

WhenResultHTTPTimePrice
3 h agoPassed402221 ms$0.02
7 h agoPassed402195 ms$0.02
11 h agoPassed402194 ms$0.02
14 h agoPassed402185 ms$0.02
19 h agoPassed402210 ms$0.02
22 h agoPassed402191 ms$0.02
25 h agoPassed402191 ms$0.02
30 h agoPassed402181 ms$0.02