1. Commerce
  2. Store Front
  3. SEO & Analytics
  4. Search Engine Optimization (SEO)
  5. SEO and Robots.txt

SEO and Robots.txt

The robots.txt file is the first file that a search engine requests when indexing your site. This file lets you tell search engines which pages on your site shouldn't be indexed. When you're first setting up your site, it's important to have your robots.txt file ready before you go live. This is especially important if you've got faceted navigation. Faceted navigation can create lots of URLs that look like they have the same content to search engines. Duplicate content can hurt your search engine ranking, so you should use the robots.txt file to control what gets indexed and stop search engines from indexing pages that seem the same. For information about creating the robots.txt file when you use faceted navigation, see Robots.txt with Categories and Facets.

Important: Test the Robots.txt File

Before you take your site live, it's important to test the robots.txt file to see how different URLs behave. The best tool for this is the Robots Testing Tool in Google Webmaster Tools.

https://www.google.com/webmasters/tools/robots-testing-tool

Sample robot.txt file.

How to Create the Robots.txt File

The robots.txt file is a text file, so you can use any text editor to create it.

Robots.txt Common Commands

The following sample robots.txt files show you some common ways to allow or block indexing.

Allow all web crawlers to crawl all content:

          User-agent: *
Disallow:

Block all web crawlers from all content:

          User-agent: *
Disallow: /

Block a specific web crawler from all content:

          User-agent: Googlebot
Disallow: /

Block a specific web crawler from a specific facet and all its values:

          User-agent: Googlebot
Disallow: /facet/*

Block all crawlers from a specific facet disregarding the order in which it appears:

          User-agent: *
Disallow: */facet/*

Allow all crawlers to crawl a specific facet value within a facet, disregarding the order in which it appears:

          User-agent: *
Disallow: */facet/*
Allow: */facet/facet-value-1

Allow all crawlers to crawl a specific facet value within a facet only when this facet appears first:

          User-agent: *
Disallow: /facet/*
Allow: /facet/facet-value-1

Block all web crawlers from adding items to cart by following “Add to Cart” links:

Note:

This is applicable only to Site Builder sites. Commerce web stores do not have “Add to Cart” links available to web crawlers.

          User-agent: *
Disallow: /additemtocart.nl

Robots.txt File Location

The robots.txt file should be in the root folder of your website. You can create the robots.txt file on your local drive and upload it to the file cabinet.

To add the robots.txt file to the file cabinet

  1. Go to Documents > Files > File Cabinet.

  2. In the file cabinet, go to Web Site Hosting files > Live Hosting Files.

  3. Click Add File.

  4. Browse to the location of your robots.txt file and select it.

  5. Click Open. This adds the file to the file cabinet.

Related Topics

General Notices