ToolVet
78trust / 100

Dev Robots For Bot

by robots-allowed-for-bot in AI models & inference

x402 APIPassing, checked 1 h ago

Is a specific bot allowed to fetch a URL? Fetches the site's robots.txt and evaluates the path for the named user-agent (GPTBot, ClaudeBot, Googlebot, Bingbot, CCBot, PerplexityBot or any token), returning allowed/disallowed, the matching rule and group, crawl-delay, and whether the bot has a dedicated group or falls back to *. Pre-flight for compliant scraping agents. $0.01 per check.

POST https://intel.rallylive.ca/dev/robots-for-bot

Last 30 days

All checks passedSome failedAll failedNot checked
Uptime
100%
Response time
49 ms typical, 298 ms slowest 5%
Last check
1 h ago
Next check
any minute now

How to call it

# See the payment challenge (nothing is charged)
curl -i -X POST "https://intel.rallylive.ca/dev/robots-for-bot" \
  -H "content-type: application/json" \
  -d '{"query":"https://www.reddit.com/r/programming | GPTBot"}'
import { wrapFetchWithPayment } from "@x402/fetch";
import { x402Client } from "@x402/core/client";
import { ExactEvmScheme } from "@x402/evm/exact/client";
import { privateKeyToAccount } from "viem/accounts";

const client = new x402Client().register(
  "eip155:8453",
  new ExactEvmScheme(privateKeyToAccount(process.env.AGENT_KEY)),
);
const pay = wrapFetchWithPayment(fetch, client);

// Not sure it's safe to pay? Preflight it first for $0.005:
// GET https://toolvet.app/api/v1/check?url=https%3A%2F%2Fintel.rallylive.ca%2Fdev%2Frobots-for-bot
const res = await pay("https://intel.rallylive.ca/dev/robots-for-bot", {
  method: "POST",
  headers: { "content-type": "application/json" },
  body: JSON.stringify({"query":"https://www.reddit.com/r/programming | GPTBot"}),
});
console.log(await res.json());

Example input

{
  "query": "https://www.reddit.com/r/programming | GPTBot"
}

Example output

{
  "allowed": false,
  "bot": "GPTBot",
  "crawl_delay": null,
  "dedicated_group": true,
  "group": "gptbot",
  "matched_rule": "disallow: /",
  "url": "https://www.reddit.com/r/programming"
}

Security scan

  • Payment address changed recently in payment
    Payment address changed from 0x91D568599fB802Cb3f03F067D2063785dBAF0abd to 0xFC24FD34da9e49A0110DCaEe3Ad30eC54175eBB3

Recent checks

WhenResultHTTPTimePrice
1 h agoPassed40257 ms$0.01
5 h agoPassed40248 ms$0.01
8 h agoPassed40249 ms$0.01
12 h agoPassed402298 ms$0.01
15 h agoPassed40245 ms$0.01
19 h agoPassed40255 ms$0.01
22 h agoPassed40276 ms$0.01
26 h agoPassed40247 ms$0.01
30 h agoPassed40251 ms$0.01
33 h agoPassed40248 ms$0.01
37 h agoPassed402111 ms$0.01
41 h agoPassed40243 ms$0.01
44 h agoPassed402567 ms$0.01
2 days agoPassed40247 ms$0.01

History

11 Oct 2026Payment address changed from 0x91D568599fB802Cb3f03F067D2063785dBAF0abd to 0xFC24FD34da9e49A0110DCaEe3Ad30eC54175eBB3