ToolVet
81trust / 100

Proxy Site Crawl

by Website Crawler to Markdown in Search & web

x402 APIPassing, checked 40 min ago

Website crawler: read a whole website or one section of it (for example every page under /docs) as clean Markdown: starts at a URL, follows same-site links breadth-first and returns up to 20 pages with title and description. Use for documentation, product catalogs or company sites. For one page use web-scrape, for a known list of URLs web-read-batch. Charged only if at least one page was read.

POST https://agentsvc.io/api/v1/proxy/site-crawl

Last 30 days

All checks passedSome failedAll failedNot checked
Uptime
100%
Response time
33 ms typical, 33 ms slowest 5%
Last check
40 min ago
Next check
any minute now

How to call it

# See the payment challenge (nothing is charged)
curl -i -X POST "https://agentsvc.io/api/v1/proxy/site-crawl" \
  -H "content-type: application/json" \
  -d '{"max_chars":2000,"max_pages":3,"url":"https://agentsvc.io/docs"}'
import { wrapFetchWithPayment } from "@x402/fetch";
import { x402Client } from "@x402/core/client";
import { ExactEvmScheme } from "@x402/evm/exact/client";
import { privateKeyToAccount } from "viem/accounts";

const client = new x402Client().register(
  "eip155:8453",
  new ExactEvmScheme(privateKeyToAccount(process.env.AGENT_KEY)),
);
const pay = wrapFetchWithPayment(fetch, client);

// Not sure it's safe to pay? Preflight it first for $0.005:
// GET https://toolvet.app/api/v1/check?url=https%3A%2F%2Fagentsvc.io%2Fapi%2Fv1%2Fproxy%2Fsite-crawl
const res = await pay("https://agentsvc.io/api/v1/proxy/site-crawl", {
  method: "POST",
  headers: { "content-type": "application/json" },
  body: JSON.stringify({"max_chars":2000,"max_pages":3,"url":"https://agentsvc.io/docs"}),
});
console.log(await res.json());

Example input

{
  "max_chars": 2000,
  "max_pages": 3,
  "url": "https://agentsvc.io/docs"
}

Example output

{
  "data": {
    "crawled_at": "2026-10-06T14:17:50.682Z",
    "discovered_urls": 37,
    "ok_count": 2,
    "pages": [
      {
        "description": "Quickstart, payment protocol (x402 v1 and v2), free trial, error codes and th...",
        "markdown": "Documentation\n\n# Pay per call. Nothing else to set up.\n\nEvery service is a JS...",
        "ok": true,
        "status": 200,
        "title": "Docs: how agents pay per call with x402 · agentsvc.io",
        "url": "https://agentsvc.io/docs",
        "word_count": 104
      }
    ],
    "pages_crawled": 2,
    "site": "https://agentsvc.io",
    "skipped_non_html": 2,
    "took_ms": 125
  },
  "success": true
}

Security scan

  • No findings. We scan names, descriptions and tool definitions for hidden instructions and other prompt-injection patterns.

Recent checks

WhenResultHTTPTimePrice
40 min agoPassed40233 ms$0.005
6 h agoPassed40236 ms$0.005