81trust / 100

Robots Check

by Robots.txt check in Search & web

x402 APIPassing, checked 4 h ago

Fetch a site's robots.txt and answer: may this user-agent crawl this path? Returns the matched rule and all declared sitemaps.

POST https://agent402.tools/api/robots-check

Last 30 days

All checks passedSome failedAll failedNot checked
Uptime
100%
Response time
185 ms typical, 185 ms slowest 5%
Last check
4 h ago
Next check
any minute now

How to call it

# See the payment challenge (nothing is charged)
curl -i -X POST "https://agent402.tools/api/robots-check" \
  -H "content-type: application/json" \
  -d '{"url":"https://agent402.tools/api/pricing","userAgent":"MyAgent"}'
import { wrapFetchWithPayment } from "@x402/fetch";
import { x402Client } from "@x402/core/client";
import { ExactEvmScheme } from "@x402/evm/exact/client";
import { privateKeyToAccount } from "viem/accounts";

const client = new x402Client().register(
  "eip155:8453",
  new ExactEvmScheme(privateKeyToAccount(process.env.AGENT_KEY)),
);
const pay = wrapFetchWithPayment(fetch, client);

// Not sure it's safe to pay? Preflight it first for $0.005:
// GET https://toolvet.app/api/v1/check?url=https%3A%2F%2Fagent402.tools%2Fapi%2Frobots-check
const res = await pay("https://agent402.tools/api/robots-check", {
  method: "POST",
  headers: { "content-type": "application/json" },
  body: JSON.stringify({"url":"https://agent402.tools/api/pricing","userAgent":"MyAgent"}),
});
console.log(await res.json());

Example input

{
  "url": "https://agent402.tools/api/pricing",
  "userAgent": "MyAgent"
}

Example output

{
  "allowed": true,
  "matchedRule": "Allow: /",
  "path": "/api/pricing",
  "sitemaps": [
    "https://agent402.tools/sitemap.xml",
    "https://agent402.tools/sitemapindex.xml"
  ],
  "userAgent": "MyAgent"
}

Security scan

  • No findings. We scan names, descriptions and tool definitions for hidden instructions and other prompt-injection patterns.

Recent checks

WhenResultHTTPTimePrice
4 h agoPassed402185 ms$0.001