78trust / 100

Monitor Robots Allowed

by crawl-allowed-check in Search & web

x402 APIPassing, checked 2 h ago

Is a URL crawlable? Fetches the site's robots.txt and evaluates the given path for a specific user-agent (Googlebot, GPTBot, ClaudeBot, Bingbot or any token; default *), returning allowed/disallowed, the matching rule, the group used, crawl-delay and declared sitemaps. Pre-flight for crawlers and SEO checks. $0.01 per check.

POST
https://intel.rallylive.ca/monitor/robots-allowed
Price
$0.01
per call in USDC
Uptime
100%
last 30 days
Response time
49 ms
193 ms slowest 5%
Last check
Passed
2 h ago, next any minute now

Run a free test callHow to call it

Reliability

All checks passedSome failedAll failedNot checked

Status and outage history · Atom feed

How to call it

POST https://intel.rallylive.ca/monitor/robots-allowed

Each call costs $0.01 in USDC on Base, paid with the request over x402.

Run a free test callOne unpaid request, the way an agent sends it. You see the payment challenge it gets back. Nothing is charged.

Example request

From the listing.

POST https://intel.rallylive.ca/monitor/robots-allowed
content-type: application/json

{
  "query": "https://github.com/features | Googlebot"
}

Example response

The provider's example. We can't check it without paying.

{
  "allowed": true,
  "crawl_delay": null,
  "group": "*",
  "matched_rule": null,
  "sitemaps": [],
  "url": "https://github.com/features",
  "user_agent": "Googlebot"
}

Call it from code

The curl command shows the payment challenge. The JavaScript and Python versions pay and call, and never pay more than $0.01.

curl: see the payment challenge
# bash or zsh: see the payment challenge. Nothing is charged.
curl -sS -g -i -X POST 'https://intel.rallylive.ca/monitor/robots-allowed' \
  -H 'content-type: application/json' \
  --data-raw '{"query":"https://github.com/features | Googlebot"}'

# x402 v2 sends the challenge base64-encoded in the PAYMENT-REQUIRED header (v1 sends it in the body).
# To read the header:
curl -sS -g -o /dev/null -D - -X POST 'https://intel.rallylive.ca/monitor/robots-allowed' \
  -H 'content-type: application/json' \
  --data-raw '{"query":"https://github.com/features | Googlebot"}' \
  | grep -i '^payment-required:' | cut -d' ' -f2 | tr -d '\r' | base64 --decode
JavaScript: pay and call
// npm i @x402/fetch @x402/evm viem
import { wrapFetchWithPayment, x402Client } from "@x402/fetch";
import { registerExactEvmScheme } from "@x402/evm/exact/client";
import { privateKeyToAccount } from "viem/accounts";

// Pays at most $0.01 per call.
const client = new x402Client().setSpendControls({ maxAmountPerPayment: "$0.01" });
registerExactEvmScheme(client, { signer: privateKeyToAccount(process.env.EVM_PRIVATE_KEY) });
const pay = wrapFetchWithPayment(fetch, client);

// Not sure it's safe to pay? ToolVet checks it first for $0.005. Pay for that from a separate
// client: this one is capped at $0.01 per payment. GET https://toolvet.app/api/v1/check?url=https%3A%2F%2Fintel.rallylive.ca%2Fmonitor%2Frobots-allowed&method=POST
const res = await pay("https://intel.rallylive.ca/monitor/robots-allowed", {
  method: "POST",
  headers: { "content-type": "application/json" },
  body: JSON.stringify({
    "query": "https://github.com/features | Googlebot"
  }),
});
console.log(res.status, await res.text());
Python: pay and call
# pip install "x402[requests,evm]"
import os
from eth_account import Account
from x402 import x402ClientSync
from x402.http.clients import x402_requests
from x402.mechanisms.evm import EthAccountSigner
from x402.mechanisms.evm.exact.register import register_exact_evm_client

# Pays at most $0.01 per call.
client = x402ClientSync().set_spend_controls({"max_amount_per_payment": "$0.01"})
register_exact_evm_client(client, EthAccountSigner(Account.from_key(os.environ["EVM_PRIVATE_KEY"])))

with x402_requests(client) as session:
    res = session.post(
        "https://intel.rallylive.ca/monitor/robots-allowed",
        json={
            "query": "https://github.com/features | Googlebot",
        },
    )
    print(res.status_code, res.text)

Security scan

1 finding

  • Payment address changed recently in payment
    Payment address changed from 0x91D568599fB802Cb3f03F067D2063785dBAF0abd to 0xFC24FD34da9e49A0110DCaEe3Ad30eC54175eBB3

Recent checks

WhenResultHTTPTimePrice
2 h agoPassed40243 ms$0.01
6 h agoPassed402538 ms$0.01
10 h agoPassed40292 ms$0.01
14 h agoPassed40268 ms$0.01
18 h agoPassed40254 ms$0.01
Show 10 older checks
22 h agoPassed402152 ms$0.01
26 h agoPassed40248 ms$0.01
30 h agoPassed40256 ms$0.01
34 h agoPassed402338 ms$0.01
37 h agoPassed40241 ms$0.01
41 h agoPassed40245 ms$0.01
44 h agoPassed40240 ms$0.01
2 days agoPassed40245 ms$0.01
2 days agoPassed40242 ms$0.01
2 days agoPassed40292 ms$0.01

History

11 Oct 2026Payment address changed from 0x91D568599fB802Cb3f03F067D2063785dBAF0abd to 0xFC24FD34da9e49A0110DCaEe3Ad30eC54175eBB3