Crawler
A crawler is a program that visits web pages automatically, following links and reading what it finds, for a search engine or another service.
- Web crawler
What it means
Google calls a crawler “a generic term for any program that is used to automatically discover and scan websites”. Crawlers are sometimes called robots or spiders.1
Google Search’s crawler is Googlebot, which Google describes as “the generic name for two types of web crawlers”, one for mobile and one for desktop.2
Crawlers matter to a shop beyond search. Merchant Center’s automatic updates use what its crawler finds on your site to update price and availability.3 Googlebot also compares the price in your feed with the price on the landing page.4
Think of it like this
A crawler works like a meter reader walking a street. It calls at each door on its own schedule, notes what it finds, and the bill is worked out later from those readings.
For example
A bath’s page says £499.00 after a price rise, but the feed still says £479.00. When Google’s crawler next visits the page, the two prices disagree, and the product may be disapproved for a mismatched price until they match.4
Why it matters
A crawler cannot read what it cannot reach. For a merchant listing, Google says “Image URLs must be crawlable and indexable”, so an image behind a dead or blocked link counts as missing.5
Not to be confused with
A crawler is not a feed. A product feed is data you send to Google; a crawler is Google fetching what your pages say, and Merchant Center compares the two.
Sources
- Google Crawling Infrastructure. Overview of Google crawlers and fetchers (user agents).
- Google Search Central. Googlebot.
- Google Merchant Center Help. Allow Merchant Center to update product information automatically.
- Google Merchant Center Help. How to fix: Mismatched product price.
- Google Search Central. Merchant listing (Product, Offer) structured data.