Robots Txt Analyze

Last 30 days

Uptime

100%

Response time

178 ms typical, 178 ms slowest 5%

Last check

46 min ago

Next check

any minute now

How to call it

# See the payment challenge (nothing is charged)
curl -i -X GET "https://netintel.dev/robots-txt/analyze?domain=example.com"

import { wrapFetchWithPayment } from "@x402/fetch";
import { x402Client } from "@x402/core/client";
import { ExactEvmScheme } from "@x402/evm/exact/client";
import { privateKeyToAccount } from "viem/accounts";

const client = new x402Client().register(
  "eip155:8453",
  new ExactEvmScheme(privateKeyToAccount(process.env.AGENT_KEY)),
);
const pay = wrapFetchWithPayment(fetch, client);

// Not sure it's safe to pay? Preflight it first for $0.005:
// GET https://toolvet.app/api/v1/check?url=https%3A%2F%2Fnetintel.dev%2Frobots-txt%2Fanalyze
const res = await pay("https://netintel.dev/robots-txt/analyze?domain=example.com");
console.log(await res.json());

Example input

{
  "domain": "example.com"
}

Example output

{
  "domain": "example.com",
  "findings": [],
  "found": true,
  "grade": "B",
  "path_check": {
    "allowed": false,
    "matched_rule": "Disallow: /private/",
    "path": "/private/page",
    "rule_type": "disallow",
    "user_agent": "*"
  },
  "raw_content_preview": "User-agent: *\nDisallow: /private/\n...",
  "robots_url": "https://example.com/robots.txt",
  "rules": [
    {
      "allow": [
        "/public/"
      ],
      "crawl_delay": null,
      "disallow": [
        "/private/",
        "/admin/"
      ],
      "user_agent": "*"
    }
  ],
  "score": 85,
  "sitemaps": [
    "https://example.com/sitemap.xml"
  ],
  "status_code": 200
}

Security scan

• No findings. We scan names, descriptions and tool definitions for hidden instructions and other prompt-injection patterns.

Recent checks