Crawler Diligence
by TariffMonkee in Search & web
AI crawler monetization check for a domain: per-crawler robots.txt rules for 18 named AI crawlers and whether access is already sold through TollBit, RSL, x402 or a 402 response. Verdict: unmonetized-crawler-magnet, already-monetized, blocked-not-monetized or low-appeal. A revenue figure appears only when the caller supplies a rate and volume.
https://tariffmonkee.com/paid/crawler-diligence- Price
- $0.04
- per call in USDC
- Uptime
- 100%
- last 30 days
- Response time
- 295 ms
- 2.1 s slowest 5%
- Last check
- Passed
- 2 h ago, next any minute now
Reliability
Status and outage history · Atom feed · Watch with other listings
How to call it
GET https://tariffmonkee.com/paid/crawler-diligence
Each call costs $0.04 in USDC on Base, paid with the request over x402.
Run a free test callOne unpaid request, the way an agent sends it. You see the payment challenge it gets back. Nothing is charged.
Example request
From the listing.
GET https://tariffmonkee.com/paid/crawler-diligence?domain=example.comParameters
| Parameter | Type | Required | Description |
|---|---|---|---|
domain | string | Yes | Bare domain to screen, no scheme. |
monthly_crawls | number | No | Your assumed monthly AI crawler hits. Required together with rate_per_crawl. |
rate_per_crawl | number | No | Your assumed USD per crawl. Required together with monthly_crawls. No rate is assumed for you: none is published industry-wide. |
Example response
The listing has no example response.
Call it from code
The curl command shows the payment challenge. The JavaScript and Python versions pay and call, and never pay more than $0.04.
curl: see the payment challenge
# bash or zsh: see the payment challenge. Nothing is charged.
curl -sS -g -i 'https://tariffmonkee.com/paid/crawler-diligence?domain=example.com'
# x402 v2 sends the challenge base64-encoded in the PAYMENT-REQUIRED header (v1 sends it in the body).
# To read the header:
curl -sS -g -o /dev/null -D - 'https://tariffmonkee.com/paid/crawler-diligence?domain=example.com' \
| grep -i '^payment-required:' | cut -d' ' -f2 | tr -d '\r' | base64 --decodeJavaScript: pay and call
// npm i @x402/fetch @x402/evm viem
import { wrapFetchWithPayment, x402Client } from "@x402/fetch";
import { registerExactEvmScheme } from "@x402/evm/exact/client";
import { privateKeyToAccount } from "viem/accounts";
// Pays at most $0.04 per call.
const client = new x402Client().setSpendControls({ maxAmountPerPayment: "$0.04" });
registerExactEvmScheme(client, { signer: privateKeyToAccount(process.env.EVM_PRIVATE_KEY) });
const pay = wrapFetchWithPayment(fetch, client);
// Not sure it's safe to pay? ToolVet checks it first for $0.005. Pay for that from a separate
// client: this one is capped at $0.04 per payment. GET https://toolvet.app/api/v1/check?url=https%3A%2F%2Ftariffmonkee.com%2Fpaid%2Fcrawler-diligence
const res = await pay("https://tariffmonkee.com/paid/crawler-diligence?domain=example.com");
console.log(res.status, await res.text());Python: pay and call
# pip install "x402[requests,evm]"
import os
from eth_account import Account
from x402 import x402ClientSync
from x402.http.clients import x402_requests
from x402.mechanisms.evm import EthAccountSigner
from x402.mechanisms.evm.exact.register import register_exact_evm_client
# Pays at most $0.04 per call.
client = x402ClientSync().set_spend_controls({"max_amount_per_payment": "$0.04"})
register_exact_evm_client(client, EthAccountSigner(Account.from_key(os.environ["EVM_PRIVATE_KEY"])))
with x402_requests(client) as session:
res = session.get("https://tariffmonkee.com/paid/crawler-diligence?domain=example.com")
print(res.status_code, res.text)Security scan
No findings
We scan names, descriptions and tool definitions for hidden instructions and other prompt-injection patterns.
Recent checks
| When | Result | HTTP | Time | Price |
|---|---|---|---|---|
| 2 h ago | Passed | 402 | 1.2 s | $0.04 |
| 6 h ago | Passed | 402 | 1.7 s | $0.04 |
| 10 h ago | Passed | 402 | 1.1 s | $0.04 |
| 14 h ago | Passed | 402 | 1.4 s | $0.04 |
| 18 h ago | Passed | 402 | 143 ms | $0.04 |
Show 10 older checks
| 22 h ago | Passed | 402 | 198 ms | $0.04 |
| 26 h ago | Passed | 402 | 77 ms | $0.04 |
| 29 h ago | Passed | 402 | 2.1 s | $0.04 |
| 32 h ago | Passed | 402 | 1.9 s | $0.04 |
| 35 h ago | Passed | 402 | 1.6 s | $0.04 |
| 38 h ago | Passed | 402 | 51 ms | $0.04 |
| 41 h ago | Passed | 402 | 80 ms | $0.04 |
| 44 h ago | Passed | 402 | 295 ms | $0.04 |
| 2 days ago | Passed | 402 | 220 ms | $0.04 |
| 2 days ago | Passed | 402 | 125 ms | $0.04 |