How search engines crawl websites: a beginner’s guide

SEO course › Part 1: SEO foundations

Here is how search engines crawl websites: they send automated programs, called crawlers or bots, to visit web pages, read their content and follow the links on them to discover more pages. Google’s crawler is Googlebot and Bing’s is Bingbot. Crawling is the first step in search: a page that is never crawled cannot appear in search results.

This lesson explains how search engines crawl websites, how they find your pages, what can block them, and the simple steps you can take to make crawling easier.

What is crawling in SEO?

Crawling is the discovery stage of search. A crawler requests a page from your server, much like your browser does, and then reads the code it receives.

Search engines work in three broad stages: crawling, indexing and ranking. Crawling finds pages, search engine indexing stores and understands them, and ranking decides the order of results. If you are new to the field, start with what SEO is and why it matters.

How do search engines find new pages?

There is no central list of every web page. Crawlers have to discover URLs (web addresses) on their own. They mainly do this in three ways.

1. Following links. When Googlebot crawls a page it already knows, it notes the links on that page and adds new URLs to its list of pages to visit.
2. Reading sitemaps. An XML sitemap is a file that lists the pages you want search engines to know about. You can submit it in Google Search Console and Bing Webmaster Tools.
3. Direct requests. You can ask Google to crawl a specific URL with the URL Inspection tool in Search Console. Bing and some other search engines also support IndexNow, a protocol that lets sites notify them when pages are added or changed.

Links remain the main route. A page with no internal links pointing to it, called an orphan page, is hard for crawlers to find.

How does Google crawl a website step by step?

1. Googlebot picks a URL from its crawl queue, a list of pages waiting to be visited.
2. It checks your robots.txt file to see whether it is allowed to crawl that URL.
3. It requests the page from your server and receives a status code, such as 200 (OK), 301 (moved permanently) or 404 (not found).
4. It reads the HTML and, where needed, renders the page by running its JavaScript, much like a browser would.
5. It extracts the links it finds and adds new URLs to the queue.

Google has said it crawls most sites with its smartphone crawler, because it mainly uses the mobile version of pages for indexing. So what your mobile pages show matters most.

What is robots.txt and how does it affect crawling?

Robots.txt is a plain text file at the root of your site, such as example.com/robots.txt. It tells crawlers which parts of the site they should not crawl.

For example, the line Disallow: /cart/ asks crawlers to stay out of your cart pages. Well-behaved crawlers like Googlebot and Bingbot follow these rules.

One important detail: robots.txt controls crawling, not indexing. Google can still index a blocked URL, without its content, if other pages link to it. To keep a page out of search results, use a noindex rule instead, and do not block that page in robots.txt, or Google will never see the rule. Google’s introduction to robots.txt explains the details.

What is crawl budget, and should you worry about it?

Crawl budget is the number of URLs a search engine can and wants to crawl on your site in a given period. It depends on how well your server copes with requests and how much demand there is for your content.

Google has said crawl budget is mainly a concern for very large sites or sites whose pages change very often. For most small business sites with a few hundred pages, it is rarely a problem.

Example: an online saree shop

Imagine a small online saree shop run from Hyderabad with 300 products. Each category page has filters for color, fabric, price and occasion.

Every filter combination creates a new URL, such as /sarees/?color=red&fabric=silk&sort=price. Suddenly, 300 products can produce thousands of near-duplicate URLs, and crawlers may spend time on these instead of new product pages.

The shop fixes this by blocking crawling of low-value filter URLs in robots.txt, linking clearly to its main category and product pages, and keeping its sitemap limited to pages it actually wants in search.

How can you make your site easier to crawl?

• Link to every important page from at least one other page on your site.
• Submit an XML sitemap and keep it up to date.
• Make sure robots.txt does not accidentally block important sections, CSS or JavaScript files.
• Fix broken links and long redirect chains (one redirect leading to another, and then another).
• Keep your server fast and reliable, since Google may crawl less if your server is slow or returns errors.
• Check the Crawl stats report and Page indexing report in Google Search Console for problems.

For more quick wins beyond crawling, see our SEO tips for beginners.

Common mistakes to avoid

• Leaving a Disallow: / rule in robots.txt after launching a site, which blocks crawling of the whole site.
• Using robots.txt to try to remove a page from Google’s results.
• Hiding important links behind JavaScript click actions instead of normal HTML links that crawlers can follow.
• Creating endless URL variations through filters, session IDs or calendar pages.

Key takeaways

• Crawlers discover pages mainly by following links and reading sitemaps.
• Crawling comes before indexing and ranking.
• Robots.txt controls crawling, not indexing.
• Crawl budget matters mostly for large or fast-changing sites.
• Good internal linking and a clean sitemap solve most crawling issues for small sites.

Frequently asked questions

How often does Google crawl my website?

It varies. Google tends to crawl popular and frequently updated pages more often and quieter pages less often, and there is no fixed schedule.

Can I make Google crawl my site faster?

You can request crawling of individual URLs with the URL Inspection tool and submit a sitemap. These help Google discover pages, but they do not guarantee immediate crawling or indexing.

Does Google follow the crawl-delay rule in robots.txt?

No. Googlebot ignores the crawl-delay rule, although Bing does support it.

Previous lesson: SEO vs SEM vs PPC: the difference between organic and paid search

Next lesson: Search engine indexing: how Google stores and understands pages