ToolVet
81trust / 100

Crawl

by Aayat AI in Productivity & SaaS

x402 APIPassing, checked 3 h ago

Turn a small website, or one section of it, into clean Markdown: starts from your URL, finds pages from the sitemap or by following links on the same site, obeys robots.txt, and returns up to 10 pages (title, Markdown, address) in one call. Great for docs, blogs and company sites. ?url=https://example.com/docs/&maxPages=10&path=/docs/

GET https://aayatai.com/crawl

Last 30 days

All checks passedSome failedAll failedNot checked
Uptime
100%
Response time
68 ms typical, 68 ms slowest 5%
Last check
3 h ago
Next check
any minute now

How to call it

# See the payment challenge (nothing is charged)
curl -i -X GET "https://aayatai.com/crawl?maxPages=2&url=https%3A%2F%2Fexample.org"
import { wrapFetchWithPayment } from "@x402/fetch";
import { x402Client } from "@x402/core/client";
import { ExactEvmScheme } from "@x402/evm/exact/client";
import { privateKeyToAccount } from "viem/accounts";

const client = new x402Client().register(
  "eip155:8453",
  new ExactEvmScheme(privateKeyToAccount(process.env.AGENT_KEY)),
);
const pay = wrapFetchWithPayment(fetch, client);

// Not sure it's safe to pay? Preflight it first for $0.03:
// GET https://toolvet.app/api/v1/check?url=https%3A%2F%2Faayatai.com%2Fcrawl
const res = await pay("https://aayatai.com/crawl?maxPages=2&url=https%3A%2F%2Fexample.org");
console.log(await res.json());

Example input

{
  "maxPages": 2,
  "url": "https://example.org"
}

Example output

{
  "complete": true,
  "count": 1,
  "crawledAt": "2026-09-28T12:00:00.000Z",
  "discoveredFrom": "links",
  "pages": [
    {
      "chars": 64,
      "markdown": "# Example Domain\n\nThis domain is for use in illustrative examples.",
      "title": "Example Domain",
      "truncated": false,
      "url": "https://example.com/"
    }
  ],
  "robots": "ok",
  "skipped": [
    {
      "reason": "robots.txt",
      "url": "https://example.com/admin"
    }
  ],
  "start": "https://example.com/"
}

Security scan

  • No findings. We scan names, descriptions and tool definitions for hidden instructions and other prompt-injection patterns.

Recent checks

WhenResultHTTPTimePrice
3 h agoPassed40268 ms$0.03