What is a Robots Meta Tag and How Do You Use It?

What if you don’t want a page to show up on Google?

SEO is the art of controlling how your site interfaces with search engines. Usually that means getting as much exposure as possible, but there may be times when you want to exclude a page from search results, or restrict the way search engines use it.

For any kind of SEO work, meta tags are often among the most useful tools in your toolkit. And one tag, known as the robots meta tag, lets you give special instructions directly to search engines—instructions which can change how (or if) your site appears in search results.

This gives you a higher level of fine-grain control over how and if your content appears on Google, Bing and other search engines.

What is a Robots Meta Tag and How Do You Use It?

 

What is the Robots Meta Tag and How Does It Work?

Meta tags are lines of html code on a web page that do not affect how the page looks to visitors, but contain extra information—”meta” data—that may be useful to search engines or coders.

The robots meta tag is a tag designed to talk directly to search engine programs (“robots”) as they crawl your web page. Search engines are constantly crawling the internet looking for new or updated content so that their search results are current and accurate. Most search engines will read a site’s robots tag and respect any special instructions it contains.

You just have to know how to write those instructions. And that’s what we’ll teach you.

 

Robots Meta vs. Robots.txt

Before we dive into the how-to, it’s important to understand our terms. In the world of web design and SEO you will also come across the term robots.txt or robots text file. Although the names are similar, the robots meta tag is a different from robots.txt. Both serve a similar purpose of informing search engines of special handling, but they work on different levels.

Robots.txt is a high-level file that gives instructions for a whole website or a whole area of a website. For example, robots.txt might tell a search engine not to index your company blog. (This would be a poor choice for SEO, by the way.)

The robots meta tag applies only to one specific page on your site. For example, it might tell search engines not to index this Tuesday’s blog post.

The robots tag gives you a finer level of control than robots.txt. We’ll only be covering the meta tag in this article, as it’s more useful for SEO purposes.

 

Why Should I Use This Meta Tag?

Usually you won’t need to use the robots meta tag. If you want to maximize your web traffic and page rank, you should generally let search engines do their work unimpeded. But there may be specific cases where it makes sense to turn them away, or restrict what they can do.

For example:

  • Limited privacy. If a page isn’t really meant for the general public, robots meta will reduce the chance of people finding it. It won’t be invisible—anyone with the link can still see it—but it will be off the beaten path.
  • A page isn’t ready for the public. Let’s say a new site feature is in beta, and you want lots of staff or select clients to be able to check it out. You could launch the page and use a robots meta tag to keep away search traffic till it’s ready. Just make sure to update your tag when you want to go live!
  • Spammy links. There are few quicker ways to hurt your page rank than linking out to spammy websites. But what if such a site really does have a great resource you want to share with your visitors? A robots meta tag can tell search engines not to follow the link, protecting your page rank.
  • Protecting images. For most businesses it’s best to let your (watermarked, attributed) images spread far and wide on the internet. But if you want to feature a picture without seeing it used elsewhere, a robots tag can tell Google not to index your images.

 

The How To Insert Robots Meta Tags

A robots meta tag looks something like this:

{code type=HTML}

{/code}

It’s always just a single line of code, and must be inserted in the header section of the page’s html. For example:

{code type=HTML}

<html>
<head>
…
<meta name="robots" content="nofollow" />
</head>
…{/code}

If you’re using WordPress.org, there are plugins that allow you to tinker with your meta tags on their own or as part of your SEO strategy without directly editing code. Meta tags do not affect how your website looks to your visitors.

What your robots tag does depends on what values you put into it.

 

Basic Values

In the example above, nofollow is the “value” of the metatag. But there are six basic values, and each one does something different:

  • index: search engines may index the page normally
  • noindex: search engines should not index the page; it won’t appear in search results
  • follow: search engine robots may “crawl” or follow all the links on the page
  • nofollow: robots should not crawl links on the page
  • archive: search engines may archive a complete copy of the page and show it in their search results (even if you take the page down)
  • noarchive: search engines should not archive the page

Note that not all search engines will respect all (or any) of the tags, and they are not required to treat your page the way you like. For example Baidu, the leading Chinese search engine, does not adhere to the “noindex” tag.

 

Advanced Tag Values & Alternatives

The six tags above will take care of most users’ needs, but there are more specialized ones as well as alternative tags used by some search engines.

Here are the other tag values, along with which search engines recognize them:

  • all: this combines the “index” and “follow” values, tell the search engine to both follow links and index the page (Google, Yandex)
  • none: this combines the “noindex” and “nofollow” values, and is the most restrictive of the basic tag values (Google, Ask, Yandex)
  • noimageindex: asks search engines not to index your images (Google)
  • unavailable_after: sets a deadline after which the page should no longer be shown in search results (used as, for example, “unavailable_after: 31-Dec-1999 23:59:59 EST”) (Google)
  • nocache: this is an alternative form of the “noarchive” value; it’s just another way of asking search engines not to archive a cached version of your page(Bing/Yahoo!)
  • nosnippet: this does the same thing as “noarchive” or “nocache” but goes a step further. Most search engines will generate a snippet about your webpage to show with it in search results; this asks them not to. (Google)
  • notranslate: asks search engines not to provide translations of the page in their search results (Google)
  • noodp: asks search engines not to use the Open Directory Project, a human-edited archive of web pages, for your page’s snippet (Google, Bing/Yahoo!)
  • noydir: prevents search engines from using the Yahoo! Directory description as the page’s descriptive snippet in the search results (obsolete since Yahoo! has been grabbed up by Bing)

 

Tips for Usage

Many site owners with no web design background may find themselves wanting to tinker with meta tags for SEO purposes. You don’t have to be an html guru to edit your robot meta tag, but you do need to use the values correctly.

Three things to bear in mind:

  1. You can use more than one value. Combining them gives you more nuanced control, and you can put them all in a single robot tag. Just separate multiple values with commas. For example: <meta name=”robots” content=”nofollow, noimageindex, notranslate” />
  2. Caps don’t matter. Other than passwords, on the internet caps almost never matter. For example, <meta name=”robots” content=”NoFollow” /> is the same thing as <meta name=”robots” content=”nofollow” />.
  3. Spaces don’t matter. Only commas are needed to separate multiple values. Spaces might make it easier for you to read, but you can also skip them. For example, <meta name=”robots” content=”nofollow,noimageindex,notranslate” /> is just fine.

 

Conflicting Tag Values

It’s possible for your tag to include conflicting values. For example:

{code type=HTML}{/code}

The “none” tag tells search engines not to follow your links, but the “follow” tag tells them they should follow your links. You should always double check your meta tags as you enter them to try to avoid this sort of situation. If it does happen, however, it’s not disastrous; it just has unpredictable results.

Most search engines have never given a public response about how they handle a conflict like this. Google has, and it always defers to the most restrictive tag: in the example above, it would ignore “follow.” Yandex has also explained its policy, and it goes with the default setting. So in this example, Yandex would crawl your links, Google would not, and what any other search engine might do is a mystery.

Similarly, if your robots.txt file conflicts with your robots meta tag, there is no definitive answer on how the two are resolved by most search engines.

 

Can I Leave Out Robots Meta Tags?

If all this is too much, you can safely leave out robots tags. You don’t have to include them, even to say “all” or “index.” All search engines, by default, will index your page and crawl your links unless you tell them not to. Using meta tags is completely optional.

However, if you run a complex website or want to keep certain content less visible, the robots meta tag can be a great way to do that. Even though it’s not fullproof, it’s respected by the most widely used English-language search engines and it’s a way you can control how they use your content.

 

We Can Help

At Everspark we encourage our clients to learn and experiment with how to manage meta tags. But if you’d rather entrust it to the professionals, our SEO staff is ready to help. Have a meta tag question? Want to use them to improve your traffic and promote your content? Give us a call at 770-481-1766 or email us today.

Article by Drew Jacob. Drew is a digital media specialist, creating attention-grabbing content for Everspark’s clients.