One of the more perplexing situations I come across in SEO consulting is having a business owner reach out to me because their site isn’t showing in search results. When I look into it, I find their website code is actually telling Google to ignore the site.
Yes, you read that correctly. The site owner thinks that it’s a search engine optimization issue but in reality it’s a simple error in the code.
It sounds far-fetched but I see it way more often than I should, so I wanted to talk about it.
Understanding the ‘NoIndex’ Meta Tag
The issue is a meta tag that can be used to tell search engines not to index the page. In other words, there is a line in the html code that tells search engines to ignore the page and not include it in search results. Here is how Google describes it:
“You can prevent a page from appearing in Google Search by including a noindex meta tag in the page’s HTML code. When Googlebot next crawls that page, Googlebot will see the noindex meta tag and will drop that page entirely from Google Search results, regardless of whether other sites link to it.”
This tag is very beneficial when used as intended. For example, you may have dedicated landing pages where you want to track specific campaign activity and don’t want organic traffic muddying the water or risk getting dinged for duplicate content. It can also be used if you have gated content that people are directed to only after they sign up. In all of these cases, you expect the page to always be blocked from showing in search results.
Implementing the tag involves putting the line below into the <head> section of your page:
<meta name=”robots” content=”noindex”>
The obvious problem with this tag is when it’s put on a page that you do want to show in search. Why would someone do that? Good question.
In at least a few of the situations I have seen this occur, the problem is tied to a redesign. When building a new version of a site, a developer may mark all the pages as “noindex” so the general public doesn’t stumble across a site before it’s ready. This is great as long as the developer removes the tag once the site goes live. Unfortunately, I have seen situations where the tag wasn’t removed and the newly designed pages are rolled out and replace the existing pages.
As Google crawls the new pages, the new “noindex” tag is discovered and Google dutifully respects the tag and starts removing the pages from search results. The site owner eventually sees organic traffic (and leads!) dry up and starts asking questions.
How to Identify the Issue
What’s perplexing is that there are cases where the developer isn’t able to resolve why organic traffic has slowed, which is why the owner has reached out to me. It may be simple miscommunication where one team member implemented the tag and then handed the project off to someone else. It might also simply be a case where the tag was simply forgotten about.
While the reason why the tag is still in place may be a mystery, identifying it is simple. The quickest and easiest way is to right click on the page and select “view page source” as shown to the right. (This is the process for Chrome. It’s similar, but slightly different, in other browsers.)
Don’t be intimidated by everything that is displayed when you see the source code. You don’t need to decipher it or even read through all of it. Simply hit ‘Ctrl+F’ to bring up the search box and type in “robots” to see if the code is on the page. If the code is on the page, you will see something like the screen shot below.
How to Remove the Noindex Meta Tag
The fix for this is straightforward – just remove the tag. Of course, depending on the content management system (CMS), this could be as simple as changing a setting in a plugin or it may require some coding changes. In the sample above, the tag is managed via the Yoast SEO plugin for a website set up on WordPress.
In this case, changing the robots meta tag to “index” from “noindex” is as simple as checking the new dropdown and updating the page.
The good news is that in most cases the site will start to be indexed quickly, especially if a sitemap is submitted via Google Search Console.