Skip to content
AdminKit

How do you verify a visitor claiming to be Googlebot, GPTBot or ClaudeBot really is?

A User-Agent header claiming to be Googlebot, GPTBot or ClaudeBot is just a string the client chose to send — nothing verifies it automatically. Google documents a reverse-DNS method to confirm its crawlers; OpenAI and Anthropic instead publish IP address lists. AdminKit's reverse DNS and DNS lookup tools run the check that actually applies to each crawler.

By AdminKit team, Netbase JSC · Updated

Key takeaways

  • A User-Agent header is text the client chose to send. A request that says “Googlebot”, “GPTBot” or “ClaudeBot” proves nothing until you check the source IP.
  • Google documents two checks for its crawlers: forward-confirmed reverse DNS (a PTR lookup, then a forward lookup that must land back on the same IP) or matching the IP against its published range files.
  • OpenAI and Anthropic document published IP lists, not reverse DNS. For their crawlers, the test is whether the source IP falls inside a listed CIDR range.
  • A PTR record alone is never enough. Whoever controls an IP block can make its reverse name say anything, so the forward step or the range list does the real work.
  • Match on a label boundary (.googlebot.com, not googlebot.com) and against the crawler you are checking, or a lookalike name or an unrelated Google Cloud address can slip through.

Why isn’t a User-Agent string proof of who sent a request?

Because nothing in HTTP checks it. AdminKit’s User Agent tool parses the raw User-Agent header your own browser sends and shows exactly what it reveals — browser, engine, OS, coarse device type. The same parsing logic applies to any request a server receives, including one claiming to be a search-engine or AI crawler. But parsing only tells you what the string says; it cannot tell you whether the client sending it is actually the crawler it claims to be, because nothing in HTTP stops a client from setting that header to arbitrary text. See what a User-Agent string reveals for how loosely-structured and client-controlled the header really is.

The HTTP specification itself expects this. RFC 9110 §10.1.5 says implementations “are encouraged not to use the product tokens of other implementations”, and adds that if a user agent masquerades as a different one, recipients can assume it wants responses tailored to the agent it named. There is no signature, no registry and no server-side check behind the value.

For most traffic this doesn’t matter — a browser lying about its own identity has little to gain. It matters a great deal for a request claiming to be “Googlebot” or “GPTBot,” because many sites give those specific claims special treatment: lighter rate limits, access past a bot-challenge page, or inclusion in analytics that get less scrutiny than everything else. Google’s Googlebot documentation warns that the header Googlebot uses “is often spoofed by other crawlers.” Verifying the claim means looking at something the requester doesn’t fully control — the IP address the request actually came from — against something the real crawler operator publishes.

What you haveWho controls itCan it verify a crawler?
User-Agent headerThe client, freelyNo
Source IP in your server logSet by the network path; hard to fake for a completed TCP connectionYes, as the input to a check
PTR record for that IPWhoever runs the reverse zone for the IP blockNot on its own
Forward A/AAAA record for the PTR nameWhoever runs that domain’s DNSYes, combined with the PTR
Vendor’s published IP range fileThe crawler operatorYes

How does Google’s reverse-DNS check work?

Google’s crawler-verification documentation describes forward-confirmed reverse DNS (FCrDNS) as the manual check, sufficient for most one-off lookups:

  1. Reverse DNS lookup. Take the source IP from your log and look up its PTR record — AdminKit’s reverse DNS tool does this directly. Confirm the returned hostname ends in googlebot.com, google.com or googleusercontent.com.
  2. Forward DNS confirmation. Take that hostname and look up its own A/AAAA record — AdminKit’s DNS lookup tool works for this. Confirm it resolves back to the exact same IP you started with.

Google’s page goes further than the three domains. It splits its traffic into three categories, each with its own reverse-DNS pattern and its own range file:

Google categoryExample of what it coversReverse DNS mask Google documentsRange file
Common crawlersGooglebotcrawl-***-***-***-***.googlebot.com or geo-crawl-***-***-***-***.geo.googlebot.comcommon-crawlers.json
Special-case crawlersAdsBotrate-limited-proxy-***-***-***-***.google.comspecial-crawlers.json
User-triggered fetchersGoogle Site Verifier***-***-***-***.gae.googleusercontent.com or google-proxy-***-***-***-***.google.comuser-triggered-fetchers.json and related files

So a request that claims to be Googlebot should come back with a googlebot.com name, not just any of the three domains. That detail matters for the spoof patterns below.

Here is what a passing check looks like at the command line. Example output (illustrative, documentation addresses and an example.com crawler domain standing in for the real one):

$ dig -x 198.51.100.25 +short
crawl-198-51-100-25.crawler.example.com.

$ dig crawl-198-51-100-25.crawler.example.com A +short
198.51.100.25

And a failing one, where the PTR name looks right but the forward lookup does not return the original address (illustrative):

$ dig -x 203.0.113.77 +short
crawl-203-0-113-77.crawler.example.com.

$ dig crawl-203-0-113-77.crawler.example.com A +short
$

An empty forward answer, or an answer with a different address, is a fail. Google’s own page shows the same two steps using the host command instead of dig; either tool works.

Why does the forward lookup matter so much?

Because the PTR record is the half an impostor can control. A PTR record is set by whoever controls the reverse-DNS zone for that IP block — which, for an IP that genuinely belongs to Google, is Google itself, but for a spoofed setup could be anyone who controls their own IP range and chooses to publish a misleading hostname there. Only the forward lookup, landing back on the original IP under a domain (googlebot.com) that only Google can publish records for, confirms the hostname and the address genuinely belong together. Skipping the second step is the most common mistake. See how to read a reverse DNS (PTR) lookup result and using reverse DNS to investigate an unfamiliar IP for more on why a PTR record alone is never sufficient on its own.

The same two-way test is written down as an algorithm in RFC 8601 §3, where mail servers call it “iprev”: take the client IP I, collect the names N its PTR query returns, resolve each name in N to the set of addresses L, and the test passes if I is in L. Two details from that section carry over to crawler checks. A PTR answer can hold several names, so check each one (RFC 8601 tells implementations to cap how many they follow). And the RFC notes that this test is not a strong authentication method on its own; it confirms the name and address are consistent, nothing more.

If you script the check, the details are easy to get wrong. A minimal sketch in Python, using only the standard library:

import ipaddress
import socket

GOOGLEBOT_SUFFIXES = (".googlebot.com",)  # the mask for common crawlers

def fcrdns_ok(ip: str, suffixes=GOOGLEBOT_SUFFIXES) -> bool:
    addr = ipaddress.ip_address(ip)
    try:
        host = socket.gethostbyaddr(ip)[0].rstrip(".").lower()
    except OSError:
        return False                      # no PTR record at all
    if not host.endswith(suffixes):        # note the leading dot
        return False
    try:
        infos = socket.getaddrinfo(host, None)
    except OSError:
        return False                      # name does not resolve forward
    forward = {ipaddress.ip_address(i[4][0].split("%")[0]) for i in infos}
    return addr in forward

Comparing ipaddress objects rather than strings avoids false mismatches between equivalent IPv6 spellings.

How do you verify GPTBot and ClaudeBot?

By matching the source IP against the operator’s published ranges. Neither OpenAI nor Anthropic documents an FCrDNS check, so assuming every major crawler supports the same method Google documents would be checking against something the operator never described:

  • OpenAI publishes a separate IP-range JSON file for each of its agents — openai.com/gptbot.json for GPTBot, plus searchbot.json for OAI-SearchBot, chatgpt-user.json for ChatGPT-User and adsbot.json for OAI-AdsBot — on its published crawler documentation, which lists a “Published IP addresses” link under each agent. It does not document a reverse-DNS verification path.
  • Anthropic takes the same approach for ClaudeBot, Claude-SearchBot and Claude-User: its support documentation states that a source IP appearing on the list at claude.com/crawling/bots.json indicates the request came from Anthropic.

Practically, this means checking a “GPTBot” or “ClaudeBot” claim correctly is a different process than checking a “Googlebot” one: fetch the vendor’s current JSON file and test whether the source IP falls inside one of its listed CIDR ranges, rather than running a PTR lookup and expecting a googlebot.com-shaped answer. AdminKit’s reverse DNS tool will still return whatever PTR record (if any) that IP happens to have, but neither OpenAI nor Anthropic has documented that record as part of how they expect their crawlers to be verified — so treat a PTR-based check alone as inconclusive for those two, not as a pass or fail.

Google publishes machine-readable range files too (the crawler-specific files in the table above, plus a general list at gstatic.com/ipranges/goog.json for other Google traffic). They are the better option for checking in bulk, though they require re-downloading the list periodically rather than a one-off lookup.

How do you check an IP against a published range file?

For a single address, AdminKit’s IP in CIDR tool does it in the browser: paste the source IP and the ranges from the vendor’s file, and it reports, range by range, whether the address is contained. Nothing you paste is sent to a server.

For log-scale checks, a script is simpler. The Google, OpenAI and Anthropic files share one shape: a creationTime field and a prefixes list in which each entry has an ipv4Prefix or ipv6Prefix key. A sketch that works on a local copy of any of them:

import ipaddress
import json

def load_ranges(path: str) -> list:
    with open(path) as f:
        data = json.load(f)
    nets = []
    for entry in data["prefixes"]:
        prefix = entry.get("ipv4Prefix") or entry.get("ipv6Prefix")
        if prefix:
            nets.append(ipaddress.ip_network(prefix))
    return nets

def ip_in_ranges(ip: str, nets: list) -> bool:
    addr = ipaddress.ip_address(ip)
    return any(addr in net for net in nets)  # False across IPv4/IPv6

ranges = load_ranges("gptbot.json")          # a copy you downloaded
print(ip_in_ranges("198.51.100.25", ranges))

The ipaddress module handles both IP versions, and testing an IPv4 address against an IPv6 network simply returns False. Re-download the files on a schedule and record the creationTime you used, so a later review can see which version of the list a decision was based on.

Which spoof patterns does each check catch?

Each check closes a different gap. The table assumes the request’s User-Agent claims to be the crawler named in the first column.

Spoof patternUser-Agent onlyPTR onlyFCrDNSPublished range file
“Googlebot” from a home or cloud IP with a generic or missing PTRPassesCatchesCatchesCatches
“Googlebot” from an IP block whose operator set its PTR to a googlebot.com-style namePassesPassesCatches: the forward lookup in googlebot.com won’t return that IPCatches
“Googlebot” with a lookalike PTR such as crawl.googlebot.com.example.netPassesPasses a substring matchCatches if you match the suffix with a leading dotCatches
“Googlebot” from a Google Cloud customer VMPassesPasses a rule that accepts any googleusercontent.com nameCan pass a suffix-only rule; caught by requiring the googlebot.com maskCatches with common-crawlers.json
“GPTBot” or “ClaudeBot” from any address outside the vendor’s listPassesInconclusiveNot the documented methodCatches

The Google Cloud row comes from Google’s own Compute Engine documentation: a VM without a custom PTR record answers reverse lookups with a default googleusercontent.com name. That name shows the address is in Google’s cloud, not that the request came from a Google crawler.

Which method applies to which crawler?

CrawlerDocumented verification methodAdminKit tool(s) that help
Googlebot (and other Google crawlers)Forward-confirmed reverse DNS, or Google’s published IP-range JSON filesReverse DNS + DNS lookup
GPTBot / OAI-SearchBot / ChatGPT-UserMatch source IP against OpenAI’s published JSON rangesIP in CIDR — paste the source IP and the ranges from OpenAI’s file
ClaudeBot / Claude-SearchBot / Claude-UserMatch source IP against Anthropic’s published JSON rangesIP in CIDR — paste the source IP and the ranges from Anthropic’s file

Whichever crawler a request claims to be, start from the source IP in your own server log — AdminKit’s User Agent tool is useful for understanding what a header reveals about an ordinary visitor, but a crawler-identity question always comes down to the IP address the request actually arrived from, checked against whatever method that crawler’s own operator documents, not the header text it chose to send.

What should you do when a claim fails?

Treat the request as an ordinary unidentified client that happens to use a crawler’s name. It should get the same rate limits, bot challenges and access rules as any other unknown visitor, not the allow-list treatment reserved for the real crawler. Adding a Disallow line to robots.txt will not stop it: RFC 9309 says plainly that robots.txt rules “are not a form of access authorization”, and a client that lies about its name has no reason to honour them.

The reverse also holds. If a request passes the check, it really did come from that operator, and blocking its IP ranges is a blunt way to opt out. Anthropic’s support article notes that blocking its IP addresses may not work reliably as an opt-out because it stops the crawler from reading your robots.txt; it asks site owners to use robots.txt rules instead.

Keep the evidence with the decision: the source IP, the PTR and forward answers, or the range file and its creationTime. If the same address turns up again, you can see what was checked and when. This verification method is part of AdminKit’s DNS cluster of DNS and reverse-DNS guides and tools.

This guide is general information, not a security audit of your specific network — see our disclaimer for what AdminKit's guidance does and does not commit to.

Frequently asked questions

Can I just trust a User-Agent that says 'Googlebot'?

No. Any HTTP client, script or browser can set its User-Agent header to any text it wants, including 'Googlebot' or the exact string a real AI crawler sends. Google's own Googlebot documentation states plainly that the User-Agent header Googlebot uses 'is often spoofed by other crawlers' — the string alone confirms nothing about who actually sent the request.

How does Google's reverse-DNS verification actually work?

Two steps, both required. First, a reverse DNS (PTR) lookup on the source IP, confirming the hostname it resolves to ends in googlebot.com, google.com or googleusercontent.com. Second — and this is the step people skip — a forward DNS lookup on that same hostname, confirming it resolves back to the original IP. The forward step matters because a PTR record is set by whoever controls the IP block; only the matching forward record, which only Google itself can publish under its own domain, confirms the two actually belong together. This is called forward-confirmed reverse DNS (FCrDNS).

Does the same reverse-DNS method work for GPTBot or ClaudeBot?

Not based on what either company documents. OpenAI's published verification method for GPTBot, OAI-SearchBot and ChatGPT-User is matching the source IP against CIDR ranges in JSON files it publishes (openai.com/gptbot.json and similar) — its documentation does not describe a reverse-DNS check. Anthropic documents the same approach for ClaudeBot: match the source IP against the list published at claude.com/crawling/bots.json. Treating either crawler's User-Agent as verified by a googlebot.com-style PTR match would be checking against a method the crawler's own operator never documented.

What should I actually do with AdminKit's tools to check a suspicious request?

Take the source IP from your server log (not the User-Agent) and run it through AdminKit's reverse DNS tool. If it's claiming to be Googlebot, confirm the PTR hostname ends in googlebot.com, google.com or googleusercontent.com, then run that hostname back through AdminKit's DNS lookup tool and confirm the A/AAAA record it returns matches the original IP. If it's claiming to be GPTBot or ClaudeBot, the reverse DNS tool only tells you whether the IP has any matching hostname at all — the actual verification is checking the IP against the vendor's published JSON list. Download the vendor's file and paste its ranges, with the source IP, into AdminKit's IP in CIDR tool, or script the same test for bulk checks.

Why would someone spoof a crawler's User-Agent in the first place?

Two common reasons. Some sites give verified search-engine crawlers special treatment — lighter rate limits, bypassing a paywall or bot-challenge page — so a scraper spoofing 'Googlebot' tries to inherit that treatment without being a real search engine. Others simply want their traffic to blend into logs that site owners don't scrutinize as closely, since 'Googlebot' and the major AI crawlers are usually on an allow-list rather than flagged for review.

Want AdminKit to watch this for you?

Free checks tell you the state today. Protective DNS and monitoring tell you when it changes.

Huy Nguyen (David), Founder and CEO of Netbase JSC

Talk to an expert

Huy Nguyen (David), Founder & CEO, Netbase JSC

Ask whether this fix applies to your setup, or get a second pair of eyes on your DNS and mail records.