Stay organized with collections Save and categorize content based on your preferences.
GooglebotGooglebot is the generic name for two types of web crawlers used by Google Search:
You can identify the subtype of Googlebot by looking at the HTTP user-agent request header in the request. However, both crawler types obey the same product token (user agent token) in robots.txt, and so you cannot selectively target either Googlebot Smartphone or Googlebot Desktop using robots.txt.
For most sites Google Search primarily indexes the mobile version of the content. As such the majority of Googlebot crawl requests will be made using the mobile crawler, and a minority using the desktop crawler.
How Googlebot accesses your siteFor most sites, Googlebot shouldn't access your site more than once every few seconds on average. However, due to delays it's possible that the rate will appear to be slightly higher over short periods. If your site is having trouble keeping up with Google's crawling requests, you can reduce the crawl rate.
Googlebot can crawl the first 15MB of an HTML file or supported text-based file. Each resource referenced in the HTML such as CSS and JavaScript is fetched separately, and each fetch is bound by the same file size limit. After the first 15MB of the file, Googlebot stops crawling and only sends the first 15MB of the file for indexing consideration. The file size limit is applied on the uncompressed data. Other Google crawlers, for example Googlebot Video and Googlebot Image, may have different limits.
When crawling from IP addresses in the US, the timezone of Googlebot is Pacific Time.
Other technical properties of Googlebot are described in the overview of Google's crawlers.
Blocking Googlebot from visiting your siteGooglebot discovers new URLs to crawl primarily from links embedded in previously crawled pages. It's almost impossible to keep a site secret by not publishing links to it. For example, as soon as someone clicks a link from your "secret" site to another site, your "secret" site URL may appear in the referrer tag and can be stored and published by the other site in its referrer log.
If you want to prevent Googlebot from crawling content on your site, you have a number of options. Remember there's a difference between crawling and indexing; blocking Googlebot from crawling a page doesn't prevent the URL of the page from appearing in search results:
noindex.Blocking Googlebot affects Google Search (including Discover and all Google Search features), as well as other products such as Google Images, Google Video, and Google News.
Verifying GooglebotBefore you decide to block Googlebot, be aware that the HTTP user-agent request header used by Googlebot is often spoofed by other crawlers. It's important to verify that a problematic request actually comes from Google. The best way to verify that a request actually comes from Googlebot is to use a reverse DNS lookup on the source IP of the request, or to match the source IP against the Googlebot IP ranges.
Except as otherwise noted, the content of this page is licensed under the Creative Commons Attribution 4.0 License, and code samples are licensed under the Apache 2.0 License. For details, see the Google Developers Site Policies. Java is a registered trademark of Oracle and/or its affiliates.
Last updated 2025-03-06 UTC.
[[["Easy to understand","easyToUnderstand","thumb-up"],["Solved my problem","solvedMyProblem","thumb-up"],["Other","otherUp","thumb-up"]],[["Missing the information I need","missingTheInformationINeed","thumb-down"],["Too complicated / too many steps","tooComplicatedTooManySteps","thumb-down"],["Out of date","outOfDate","thumb-down"],["Samples / code issue","samplesCodeIssue","thumb-down"],["Other","otherDown","thumb-down"]],["Last updated 2025-03-06 UTC."],[[["Googlebot is the name for Google's web crawlers, which include Googlebot Smartphone and Googlebot Desktop, used to understand and index website content."],["Googlebot primarily crawls and indexes the mobile version of websites, reflecting the dominance of mobile browsing."],["Website owners can control Googlebot's access by using robots.txt to manage crawl rate and prevent crawling of specific content."],["While blocking Googlebot prevents crawling, it doesn't automatically remove a page from Google Search results; `noindex` should be used for that purpose."],["It's crucial to verify the authenticity of Googlebot requests as its user agent is frequently imitated by other crawlers."]]],["Googlebot, comprising Desktop and Smartphone crawlers, indexes web content, primarily favoring the mobile version. It crawls most sites at a rate of once every few seconds, fetching up to 15MB of HTML or text-based files and their resources. To manage Googlebot's access, sites can use `robots.txt` to block crawling or `noindex` to prevent indexing. Blocking crawling affects Google Search and related products. Verify Googlebot requests via reverse DNS lookup or by checking the IP range.\n"]]
RetroSearch is an open source project built by @garambo | Open a GitHub Issue
Search and Browse the WWW like it's 1997 | Search results from DuckDuckGo
HTML:
3.2
| Encoding:
UTF-8
| Version:
0.7.4