Crawl Check
by Crawl Pre-Flight in Search & web
Crawl pre-flight for a domain: will you be blocked, is content behind a paywall, and how much is actually there. Returns per-bot robots.txt verdicts for 16 AI crawlers, crawlable page count from sitemaps, detected CDN and paid-access signals (x402, TollBit), plus a fetch/skip recommendation. Call before spending requests on an unknown domain.
https://crawl-preflight.postnov01.workers.dev/crawl-check/:domain- Price
- $0.02
- per call in USDC
- Uptime
- 0.0%
- last 30 days
- Response time
- -
- no timed checks yet
- Last check
- Failed
- 2 h ago, next any minute now
Reliability
Latest check failed 2 h ago: Expected HTTP 402, got 400
Status and outage history · Atom feed · Watch with other listings
How to call it
GET https://crawl-preflight.postnov01.workers.dev/crawl-check/:domain
Each call costs $0.02 in USDC on Polygon, paid with the request over x402.
Run a free test callOne unpaid request, the way an agent sends it. You see the payment challenge it gets back. Nothing is charged.
Example request
From the listing.
GET https://crawl-preflight.postnov01.workers.dev/crawl-check/:domainExample response
The provider's example. We can't check it without paying.
{
"crawl_surface": {
"is_estimate": true,
"pages": 28400,
"sitemaps_found": 2
},
"domain": "example.com",
"infrastructure": {
"cdn": [
"Cloudflare"
],
"paid_access": []
},
"reason": "No AI crawler restrictions declared.",
"recommendation": "fetch",
"robots": {
"ai_crawlers_allowed": 16,
"ai_crawlers_blocked": [],
"blocks_all_agents": false,
"present": true
}
}Call it from code
The curl command shows the payment challenge. The JavaScript and Python versions pay and call, and never pay more than $0.02.
curl: see the payment challenge
# bash or zsh: see the payment challenge. Nothing is charged.
curl -sS -g -i 'https://crawl-preflight.postnov01.workers.dev/crawl-check/:domain'
# x402 v2 sends the challenge base64-encoded in the PAYMENT-REQUIRED header (v1 sends it in the body).
# To read the header:
curl -sS -g -o /dev/null -D - 'https://crawl-preflight.postnov01.workers.dev/crawl-check/:domain' \
| grep -i '^payment-required:' | cut -d' ' -f2 | tr -d '\r' | base64 --decodeJavaScript: pay and call
// npm i @x402/fetch @x402/evm viem
import { wrapFetchWithPayment, x402Client } from "@x402/fetch";
import { registerExactEvmScheme } from "@x402/evm/exact/client";
import { privateKeyToAccount } from "viem/accounts";
// Pays at most $0.02 per call.
const client = new x402Client().setSpendControls({ maxAmountPerPayment: "$0.02" });
registerExactEvmScheme(client, { signer: privateKeyToAccount(process.env.EVM_PRIVATE_KEY) });
const pay = wrapFetchWithPayment(fetch, client);
// Not sure it's safe to pay? ToolVet checks it first for $0.005. Pay for that from a separate
// client: this one is capped at $0.02 per payment. GET https://toolvet.app/api/v1/check?url=https%3A%2F%2Fcrawl-preflight.postnov01.workers.dev%2Fcrawl-check%2F%3Adomain
const res = await pay("https://crawl-preflight.postnov01.workers.dev/crawl-check/:domain");
console.log(res.status, await res.text());Python: pay and call
# pip install "x402[requests,evm]"
import os
from eth_account import Account
from x402 import x402ClientSync
from x402.http.clients import x402_requests
from x402.mechanisms.evm import EthAccountSigner
from x402.mechanisms.evm.exact.register import register_exact_evm_client
# Pays at most $0.02 per call.
client = x402ClientSync().set_spend_controls({"max_amount_per_payment": "$0.02"})
register_exact_evm_client(client, EthAccountSigner(Account.from_key(os.environ["EVM_PRIVATE_KEY"])))
with x402_requests(client) as session:
res = session.get("https://crawl-preflight.postnov01.workers.dev/crawl-check/:domain")
print(res.status_code, res.text)Security scan
No findings
We scan names, descriptions and tool definitions for hidden instructions and other prompt-injection patterns.
Recent checks
| When | Result | HTTP | Time | Price |
|---|---|---|---|---|
| 2 h ago | Expected HTTP 402, got 400 | 400 | 474 ms | - |
| 6 h ago | Expected HTTP 402, got 400 | 400 | 449 ms | - |
| 10 h ago | Expected HTTP 402, got 400 | 400 | 457 ms | - |
| 13 h ago | Expected HTTP 402, got 400 | 400 | 648 ms | - |
| 17 h ago | Expected HTTP 402, got 400 | 400 | 345 ms | - |
Show 7 older checks
| 21 h ago | Expected HTTP 402, got 400 | 400 | 320 ms | - |
| 24 h ago | Expected HTTP 402, got 400 | 400 | 585 ms | - |
| 29 h ago | Expected HTTP 402, got 400 | 400 | 496 ms | - |
| 35 h ago | Expected HTTP 402, got 400 | 400 | 385 ms | - |
| 40 h ago | Expected HTTP 402, got 400 | 400 | 310 ms | - |
| 46 h ago | Expected HTTP 402, got 400 | 400 | 326 ms | - |
| 2 days ago | Expected HTTP 402, got 400 | 400 | 562 ms | - |