ToolVet
80trust / 100

Robots

by Tanod sitepeek in Search & web

x402 APIPassing, checked 4 h ago

sitepeek (Tanod) robots.txt check: whether a crawler may fetch a URL under RFC 9309, with the matching rule, crawl delay and sitemaps. Read-only.

POST https://tanod.dev/v1/robots

Last 30 days

All checks passedSome failedAll failedNot checked
Uptime
100%
Response time
274 ms typical, 854 ms slowest 5%
Last check
4 h ago
Next check
any minute now

How to call it

POST https://tanod.dev/v1/robots

Each call costs $0.001 in USDC on Base, paid with the request over x402.

Run a free test callOne unpaid request, the way an agent sends it. You see the payment challenge it gets back. Nothing is charged.

Example request

From the listing.

POST https://tanod.dev/v1/robots
content-type: application/json

{
  "url": "https://www.google.com/search",
  "user_agent": "Googlebot"
}

Parameters

ParameterTypeRequiredDescription
urlstringYesPublic http(s) URL to test; <origin>/robots.txt is fetched and the URL's path is matched against it.
user_agentstringNoProduct token (e.g. Googlebot) or a full User-Agent string; '*' (default) = any crawler. Default: "*".

Example response

The provider's example. We can't check it without paying.

Example response, 30 lines
{
  "allowed": false,
  "crawl_delay": null,
  "fetch": "ok",
  "group": "*",
  "http_status": 200,
  "matched_rule": {
    "line": 3,
    "pattern": "/search",
    "type": "disallow"
  },
  "path": "/search",
  "product_token": "googlebot",
  "reason": "most specific matching rule: disallow /search",
  "robots_url": "https://www.google.com/robots.txt",
  "sitemaps": [
    "https://www.google.com/sitemap.xml"
  ],
  "sitemaps_total": 1,
  "stats": {
    "groups": 5,
    "ignored_lines": 0,
    "lines": 264,
    "rules": 245
  },
  "truncated": false,
  "untrusted_content": true,
  "url": "https://www.google.com/search",
  "user_agent": "Googlebot"
}

Call it from code

The curl command shows the payment challenge. The JavaScript and Python versions pay and call, and never pay more than $0.001.

curl: see the payment challenge
# bash or zsh: see the payment challenge. Nothing is charged.
curl -sS -g -i -X POST 'https://tanod.dev/v1/robots' \
  -H 'content-type: application/json' \
  --data-raw '{"url":"https://www.google.com/search","user_agent":"Googlebot"}'

# x402 v2 sends the challenge base64-encoded in the PAYMENT-REQUIRED header (v1 sends it in the body).
# To read the header:
curl -sS -g -o /dev/null -D - -X POST 'https://tanod.dev/v1/robots' \
  -H 'content-type: application/json' \
  --data-raw '{"url":"https://www.google.com/search","user_agent":"Googlebot"}' \
  | grep -i '^payment-required:' | cut -d' ' -f2 | tr -d '\r' | base64 --decode
JavaScript: pay and call
// npm i @x402/fetch @x402/evm viem
import { wrapFetchWithPayment, x402Client } from "@x402/fetch";
import { registerExactEvmScheme } from "@x402/evm/exact/client";
import { privateKeyToAccount } from "viem/accounts";

// Pays at most $0.001 per call.
const client = new x402Client().setSpendControls({ maxAmountPerPayment: "$0.001" });
registerExactEvmScheme(client, { signer: privateKeyToAccount(process.env.EVM_PRIVATE_KEY) });
const pay = wrapFetchWithPayment(fetch, client);

// Not sure it's safe to pay? ToolVet checks it first for $0.005. Pay for that from a separate
// client: this one is capped at $0.001 per payment. GET https://toolvet.app/api/v1/check?url=https%3A%2F%2Ftanod.dev%2Fv1%2Frobots&method=POST
const res = await pay("https://tanod.dev/v1/robots", {
  method: "POST",
  headers: { "content-type": "application/json" },
  body: JSON.stringify({
    "url": "https://www.google.com/search",
    "user_agent": "Googlebot"
  }),
});
console.log(res.status, await res.text());
Python: pay and call
# pip install "x402[requests,evm]"
import os
from eth_account import Account
from x402 import x402ClientSync
from x402.http.clients import x402_requests
from x402.mechanisms.evm import EthAccountSigner
from x402.mechanisms.evm.exact.register import register_exact_evm_client

# Pays at most $0.001 per call.
client = x402ClientSync().set_spend_controls({"max_amount_per_payment": "$0.001"})
register_exact_evm_client(client, EthAccountSigner(Account.from_key(os.environ["EVM_PRIVATE_KEY"])))

with x402_requests(client) as session:
    res = session.post(
        "https://tanod.dev/v1/robots",
        json={
            "url": "https://www.google.com/search",
            "user_agent": "Googlebot",
        },
    )
    print(res.status_code, res.text)

Security scan

  • No findings. We scan names, descriptions and tool definitions for hidden instructions and other prompt-injection patterns.

Recent checks

WhenResultHTTPTimePrice
4 h agoPassed402274 ms$0.001
8 h agoPassed402854 ms$0.001
11 h agoPassed402273 ms$0.001
15 h agoPassed402600 ms$0.001
19 h agoPassed402268 ms$0.001
23 h agoPassed402275 ms$0.001
26 h agoPassed402484 ms$0.001
30 h agoPassed4022.3 s$0.001
33 h agoPassed402371 ms$0.001
37 h agoPassed402281 ms$0.001
41 h agoPassed402391 ms$0.001
44 h agoPassed402716 ms$0.001
2 days agoPassed402272 ms$0.001
2 days agoPassed402273 ms$0.001
2 days agoPassed402261 ms$0.001