ToolVet
81trust / 100

Robots

by Check robots.txt crawl rules in Search & web

x402 APIPassing, checked 3 h ago

Fetch a site's robots.txt and evaluate crawl permissions: given a URL (plus optional extra paths) and a user-agent, return whether each path is allowed or disallowed, the rule that matched, the user-agent group, the crawl-delay and any declared sitemaps. Implements the Robots Exclusion Protocol (RFC 9309) with longest-match-wins, Allow-over-Disallow tie-breaking and * / $ wildcards.

POST https://web.openverbs.com/v1/robots

Last 30 days

All checks passedSome failedAll failedNot checked
Uptime
100%
Response time
163 ms typical, 163 ms slowest 5%
Last check
3 h ago
Next check
any minute now

How to call it

# See the payment challenge (nothing is charged)
curl -i -X POST "https://web.openverbs.com/v1/robots" \
  -H "content-type: application/json" \
  -d '{"paths":["example"],"url":"https://example.com","userAgent":"example"}'
import { wrapFetchWithPayment } from "@x402/fetch";
import { x402Client } from "@x402/core/client";
import { ExactEvmScheme } from "@x402/evm/exact/client";
import { privateKeyToAccount } from "viem/accounts";

const client = new x402Client().register(
  "eip155:8453",
  new ExactEvmScheme(privateKeyToAccount(process.env.AGENT_KEY)),
);
const pay = wrapFetchWithPayment(fetch, client);

// Not sure it's safe to pay? Preflight it first for $0.03:
// GET https://toolvet.app/api/v1/check?url=https%3A%2F%2Fweb.openverbs.com%2Fv1%2Frobots
const res = await pay("https://web.openverbs.com/v1/robots", {
  method: "POST",
  headers: { "content-type": "application/json" },
  body: JSON.stringify({"paths":["example"],"url":"https://example.com","userAgent":"example"}),
});
console.log(await res.json());

Example input

{
  "paths": [
    "example"
  ],
  "url": "https://example.com",
  "userAgent": "example"
}

Security scan

  • No findings. We scan names, descriptions and tool definitions for hidden instructions and other prompt-injection patterns.

Recent checks

WhenResultHTTPTimePrice
3 h agoPassed402163 ms$0.006