13trust / 100

Crawl Check

by Crawl Pre-Flight in Search & web

x402 APIFailing, checked 2 h ago

Crawl pre-flight for a domain: will you be blocked, is content behind a paywall, and how much is actually there. Returns per-bot robots.txt verdicts for 16 AI crawlers, crawlable page count from sitemaps, detected CDN and paid-access signals (x402, TollBit), plus a fetch/skip recommendation. Call before spending requests on an unknown domain.

GET
https://crawl-preflight.postnov01.workers.dev/crawl-check/:domain
Price
$0.02
per call in USDC
Uptime
0.0%
last 30 days
Response time
-
no timed checks yet
Last check
Failed
2 h ago, next any minute now

Run a free test callHow to call it

Reliability

All checks passedSome failedAll failedNot checked

Latest check failed 2 h ago: Expected HTTP 402, got 400

Status and outage history · Atom feed · Watch with other listings

How to call it

GET https://crawl-preflight.postnov01.workers.dev/crawl-check/:domain

Each call costs $0.02 in USDC on Polygon, paid with the request over x402.

Run a free test callOne unpaid request, the way an agent sends it. You see the payment challenge it gets back. Nothing is charged.

Example request

From the listing.

GET https://crawl-preflight.postnov01.workers.dev/crawl-check/:domain

Example response

The provider's example. We can't check it without paying.

{
  "crawl_surface": {
    "is_estimate": true,
    "pages": 28400,
    "sitemaps_found": 2
  },
  "domain": "example.com",
  "infrastructure": {
    "cdn": [
      "Cloudflare"
    ],
    "paid_access": []
  },
  "reason": "No AI crawler restrictions declared.",
  "recommendation": "fetch",
  "robots": {
    "ai_crawlers_allowed": 16,
    "ai_crawlers_blocked": [],
    "blocks_all_agents": false,
    "present": true
  }
}

Call it from code

The curl command shows the payment challenge. The JavaScript and Python versions pay and call, and never pay more than $0.02.

curl: see the payment challenge
# bash or zsh: see the payment challenge. Nothing is charged.
curl -sS -g -i 'https://crawl-preflight.postnov01.workers.dev/crawl-check/:domain'

# x402 v2 sends the challenge base64-encoded in the PAYMENT-REQUIRED header (v1 sends it in the body).
# To read the header:
curl -sS -g -o /dev/null -D - 'https://crawl-preflight.postnov01.workers.dev/crawl-check/:domain' \
  | grep -i '^payment-required:' | cut -d' ' -f2 | tr -d '\r' | base64 --decode
JavaScript: pay and call
// npm i @x402/fetch @x402/evm viem
import { wrapFetchWithPayment, x402Client } from "@x402/fetch";
import { registerExactEvmScheme } from "@x402/evm/exact/client";
import { privateKeyToAccount } from "viem/accounts";

// Pays at most $0.02 per call.
const client = new x402Client().setSpendControls({ maxAmountPerPayment: "$0.02" });
registerExactEvmScheme(client, { signer: privateKeyToAccount(process.env.EVM_PRIVATE_KEY) });
const pay = wrapFetchWithPayment(fetch, client);

// Not sure it's safe to pay? ToolVet checks it first for $0.005. Pay for that from a separate
// client: this one is capped at $0.02 per payment. GET https://toolvet.app/api/v1/check?url=https%3A%2F%2Fcrawl-preflight.postnov01.workers.dev%2Fcrawl-check%2F%3Adomain
const res = await pay("https://crawl-preflight.postnov01.workers.dev/crawl-check/:domain");
console.log(res.status, await res.text());
Python: pay and call
# pip install "x402[requests,evm]"
import os
from eth_account import Account
from x402 import x402ClientSync
from x402.http.clients import x402_requests
from x402.mechanisms.evm import EthAccountSigner
from x402.mechanisms.evm.exact.register import register_exact_evm_client

# Pays at most $0.02 per call.
client = x402ClientSync().set_spend_controls({"max_amount_per_payment": "$0.02"})
register_exact_evm_client(client, EthAccountSigner(Account.from_key(os.environ["EVM_PRIVATE_KEY"])))

with x402_requests(client) as session:
    res = session.get("https://crawl-preflight.postnov01.workers.dev/crawl-check/:domain")
    print(res.status_code, res.text)

Security scan

No findings

We scan names, descriptions and tool definitions for hidden instructions and other prompt-injection patterns.

Recent checks

WhenResultHTTPTimePrice
2 h agoExpected HTTP 402, got 400400474 ms-
6 h agoExpected HTTP 402, got 400400449 ms-
10 h agoExpected HTTP 402, got 400400457 ms-
13 h agoExpected HTTP 402, got 400400648 ms-
17 h agoExpected HTTP 402, got 400400345 ms-
Show 7 older checks
21 h agoExpected HTTP 402, got 400400320 ms-
24 h agoExpected HTTP 402, got 400400585 ms-
29 h agoExpected HTTP 402, got 400400496 ms-
35 h agoExpected HTTP 402, got 400400385 ms-
40 h agoExpected HTTP 402, got 400400310 ms-
46 h agoExpected HTTP 402, got 400400326 ms-
2 days agoExpected HTTP 402, got 400400562 ms-