How to De-Index Your Website?

Have you ever experienced the unsettling realisation that something crucial was left unprotected? Imagine discovering your valuable digital content — similar to leaving your home's front door wide open — is unintentionally exposed to the world via Google. This scenario, akin to finding that your staging environment or privacy sensitive information has been indexed by Google, can spark immediate concern over who else might have gained access. In the digital realm, ensuring our content is found by the right audience on search engines like Google is a common goal. However, the necessity to conceal specific pages from public view arises more often than one might anticipate. Whether it's to mitigate legal risks, protect sensitive information, or manage outdated content, the urgency to remove undesired pages from Google is a critical task that demands immediate and careful attention.

Sections:

1. Understanding de-indexing
2. Preparing for de-indexing
3. Methods to de-index your website
4. Common use cases for de-indexing
5. Verifying de-indexing
6. Considerations and consequences

1. Understanding de-indexing

In the digital age, visibility on search engines like Google can be a double-edged sword. While it drives traffic and engagement to your website, certain circumstances necessitate the opposite—making specific content invisible to search engines. This process, known as de-indexing, involves instructing search engines to remove specific pages from their results, effectively making them inaccessible to the average online searcher. But why is de-indexing important, and how does it differ from the often-confused concept of delisting?

De-indexing

De-indexing is crucial for several reasons: protecting sensitive information, complying with legal requirements, managing your online reputation, or simply correcting the accidental exposure of non-public domains like staging sites. It gives website owners control over their content's visibility and ensures that only relevant, appropriate material is accessible through search engines.

Delisting

On the other hand, delisting refers to the removal of a website or web page from a search engine's index at the search engine's discretion, often due to violations of guidelines or the presence of harmful content. Delisting can be seen as a punitive action taken by the search engine, whereas de-indexing is a voluntary action initiated by the website owner for strategic reasons.

Understanding the distinction between these two actions is pivotal for digital professionals. De-indexing is a proactive measure, a tool in your digital toolkit for managing how your content is perceived and accessed online. Whether you're safeguarding proprietary information, mitigating legal risks, or simply ensuring that only the most relevant pages represent your brand, knowing how to navigate the nuances of de-indexing is an indispensable skill in today’s digital landscape.

2. Preparing for de-indexing

Before embarking on the de-indexing journey, a strategic pause is essential. This preparatory phase is not merely about technical readiness but also about understanding the potential impacts and ensuring that every step taken aligns with your broader digital strategy. Assessing the impact of de-indexing is the first critical step. It involves weighing the benefits of removing a page from search results against the potential drawbacks, such as reduced web traffic or diminished online visibility.

Consider the following questions to guide your impact assessment:

• What is the purpose of de-indexing this content? Identify the core reasons behind the need to de-index, whether they're related to privacy concerns, legal compliance, or brand management.
• How will de-indexing affect my website's overall traffic and search engine ranking? Understand that removing pages from search results might lead to a temporary dip in traffic. It's crucial to analyze whether the pages in question are significant traffic drivers and how their absence might affect your site's SEO performance.
• Is there a possibility of re-indexing these pages in the future? Sometimes, de-indexing is a temporary measure. Consider the long-term strategy for the content you're removing. Will these pages need to be re-introduced? If so, plan for a seamless re-indexing process.

Equally important is the consideration of alternative strategies before proceeding. In some cases, content modification or updating privacy settings may serve your objectives more effectively than full de-indexing. For instance, if the goal is to refresh outdated content, updating the existing pages with current information might be more beneficial for maintaining SEO rankings while ensuring content relevance.

Furthermore, evaluate the technical and administrative aspects of de-indexing. Ensure that you have the necessary access to implement changes such as adding no-index tags or updating the robots.txt file. For websites with multiple administrators or complex content management systems, clear communication and coordination are paramount to avoid conflicting actions or missteps.

3. Methods to de-index your website

Successfully removing web pages from search engine indexes requires a clear understanding of the available tools and techniques. Whether you have access to Google Search Console or not, several methods can be employed to achieve your de-indexing goals.

Robots.txt File

The robots.txt file plays a pivotal role in managing how search engines interact with your site. It's important to note that while the robots.txt file can prevent search engines from crawling new pages, it does not remove already indexed pages.

To prevent future indexing:

Placement: Ensure the robots.txt file is located in the root directory of your site.

/ (root)
|
|-- robots.txt  # This file should be in the root directory
|
|-- index.html  # Main HTML file for the website
|
|-- css/        # Directory for CSS files
|   |-- style.css
|   |-- layout.css
|
|-- js/         # Directory for JavaScript files
|   |-- app.js
|   |-- helper.js
|
|-- images/     # Directory for images
|   |-- logo.png
|   |-- banner.jpg

Syntax: Use the disallow directive to specify which URLs or directories should not be crawled. For example, Disallow: /private-directory/ will prevent search engines from accessing and indexing content in that directory.

// robots.txt
User-agent: *
Disallow: /private-directory/

You can debug your robots.txt file within Google Search Console or one of the robots.txt testing tools you find online. These will help you in creating the right set of rules to incorporate in your file. Which will ensure certain pages and directories won’t be indexed in the future.

Meta Tags

Utilising no-index meta tags is a powerful method for instructing search engines to exclude specific pages from their indexes. This approach is suitable for nearly all web content and does not require access to Google Search Console.

Implementation: Insert <meta name="robots" content="noindex"> into the <head> section of the HTML of each page you wish to de-index.

<head>
...
<meta name="robots" content="noindex">
...
</head>

Versatility: Meta tags are effective for a wide range of content types and offer a straightforward solution for page-specific de-indexing.

Google Search Console

For those with access to Google Search Console, this tool provides a direct method to request the removal of pages from Google's index.

Verification: Ensure your site is verified within Google Search Console. Removal Request: Use the "Remove URLs" feature to request the de-indexing of specific pages. This tool should be used with caution, as it offers a quick but temporary removal (up to 90 days), primarily for urgent situations.

Server-Side Methods

The X-Robots-Tag HTTP header is particularly useful for content types that cannot include meta tags, such as PDF files or images. This method is extremely powerful and easy to implement (when you have access) in removing content from the search engine in bulk. As we’ll see later when we talk about staging environments being indexed. This header is useful to de-index the whole environment.

Configuration: Configure your web server to include X-Robots-Tag: noindex in the HTTP header of the response for the resources you wish to de-index. Flexibility: This method allows for the control of indexing across a variety of content types, offering a versatile solution for web administrators.

Legal and Privacy Requests

In situations involving sensitive information or legal issues, more immediate action may be necessary.

Direct Requests: Search engines typically provide mechanisms to request the removal of content under certain conditions, such as copyright infringement or the presence of personal information. Documentation: Be prepared to provide detailed documentation supporting your request, adhering to the specific guidelines set by the search engine.

4. Common Use Cases for De-indexing

Alright, let's summarise what we have so far:

De-indexing differs from delisting. Delisting occurs when your pages are removed from the index due to a penalty by Google. In contrast, de-indexing is a voluntary action you take to remove pages from the index that should not be there.

It's crucial to understand the potential impact of de-indexing, especially how it might affect the organic traffic coming to your website.

There are a few methods you can employ to de-index a page from Google, such as adding a meta tag or utilising the removal tool in Google Search Console.

Fortunately, there are typical scenarios where specific methods or combinations of them are needed for effective de-indexing from Google. Below, you'll find various use-cases that will guide you in deciding the appropriate actions for your particular situation. While your circumstances might vary slightly, these use-cases generally provide a reliable blueprint to follow.

Acceptance or staging environment got indexed

Whenever you google your site and you find that your acceptance environment got indexed you need to be careful not to make the mistake of adding the Disallow: / directive to the robots.txt file of the acceptance environment. There are two reasons this is not a good idea.

1. The first being, as I wrote before, that the robots.txt file will only prevent future indexing. Meaning that everything that has been indexed will remain in the index and visible in Google.
2. The second being that the acceptance environment is often using the same repository as your production environment. This means that the code is shared, only the content will differ. That means that the robots.txt file for acceptance and production will be the same file. Putting this on Disallow: / will also block access to production for Google, something you don’t want to do.

So what do you need to do?

1. First we need to make sure that the indexed acceptance-pages are de-indexed. The best way we can do this is through the X-Robots-tag in the response of the page, as this can be set up for the entire subdomain and will affect all pages at once. If you aren’t able to do that you can resort to the no-index tag. This is a page directive so you might need some additional configuration there for different page clusters or subdirectories. And again be cautious that this won’t affect your production environment.
2. Now once that is done, we need to wait, unfortunately, until Google has removed all pages from its index. This takes time as Google needs to revisit all the indexed pages and de-index those. Given the amount of pages that have been indexed this can take up months. However when it’s done removing all of your acceptance urls, you’re basically done.

One thing to consider, when the robots.txt-file on acceptance and production aren't shared, is to add the Disallow: / directive for some additional security/safety.

Obsolete or outdated content

When content on your platform is no longer needed, but it is still indexed there are multiple things you can do.

1. The first and very simple one is to just delete the content. Anyone accessing a page that has been deleted will end up on a 404-page telling that the page no longer exists.
2. This is not very good user experience so we often redirect these kinds of pages to a parent directory. As an example /blog/old-blog would get a redirect towards /blog. By doing this we expect that the ‘value’ a page has gathered in the search engine over the timespan that it was live, is spilled over to the parent-directory, not losing any of that hard earned ‘value’. Next to that the user doesn’t end up on a 404-page when clicking on this page in the search results. The redirect, or 404 response, will tell the search engine that the page has been altered (either moved or removed) and that the new destination can be found at the end of the redirect. Given the chosen option the entry will be removed from the search engine.

One thing to consider when removing pages and redirecting them is that those pages can hold some value. Maybe not for you, but maybe for others. So the approach you take needs to be carefully considered. As I said before you can also choose to keep the page and update the content. Or if you have a new page that is going to replace the old page, you could use a canonical link element to point to that new page from the old page. The thing is that if you find yourself in a situation where you want to delete a page, you need to be very sure that you’re never going to need it anymore and by deleting it you are fully aware of what the consequences will be.

Sensitive Information Exposed

If there are pages indexed that hold sensitive information it is very important to take swift action. It’s important that there are two factors at play here.

1. One being the link to the page indexed by google.
2. The other being the actual page with sensitive information that you control.

The first thing that you want to do is to get the information offline. The action you take depends on the type of sensitive information and what the use case was in the first place. If the sensitive information should only be viewed by logged in users, it should be put behind a login wall. If it shouldn’t be out there in the first place, you can remove it. Given the content, you have the following options:

• Remove the page (404 or 410)
• Update the content on the page
• Add the no-index metatag to the page (less secure)

At the same time you also want to remove the link (entry) from the index of the search engine. This can be done manually from Google Search Console and this will last for about 6 months. In that time you need to have taken the proper action to protect the sensitive information. Because after that period the result can be shown in the search results again. If you want to show the result sooner than that, because you cleaned up the page or it is behind a login-wall, you can remove your request from Google Search Console and things will be back to normal again.

Duplicate Content Issues

First things first, duplicate content will not give you a penalty. But it's not good to have duplicate content on your website. Because it will eat up some of your crawl budget. Plus the search engine will decide which one of the two pages ends up in the search engine. So you’re also losing control on what you want to show in the index.

There can be a reason you want to have duplicate content on your website. Think of a vanity URL for marketing purposes or products with the same product text. In that case you do want the pages to exist for the user, but you don’t want them to be indexed.

Here you’ve got a couple of options:

1. The first one has been discussed already and that is to no-index the page that you don’t want in the index. This can be done by adding the noindex tag to the <head> of that page.
2. Another option you have is to use a canonical link element. We already briefly touched upon this. But in the context of duplicate content it is the easiest to explain. You can see a canonical as a redirect under the hood. Which doesn’t affect what the user sees, but does affect what Google can see. You also add it to the <head> of the page, where you specify the page that it should index. So with a canonical, a user can still visit and see the duplicate page. But whenever a bot lands on the page to index it, the canonical tells the bot that instead of this page, it should index or visit the page specified in the canonical. Here the benefit is that you’re actively addressing your duplicate content issue as you’re directly referring to the page that should be indexed. If you would only noindex the page there is no hint or sign to the actual page that you want to be indexed.

Can you use them both? You can, but the noindex takes precedence over the canonical and this might give conflicts. So my advice would be to pick one based on the goal you want to achieve.

There can also be a reason you don't want to have duplicate content on your website. Which should not be in the index and not be visible to a user. An example could be when both the http and the https version of your website have been indexed. Or when a trailing slash is not redirected. For example when /blog/post-about-something and /blog/post-about-something/ are both accessible. There is a minor difference which is the trailing slash at the end of the second example. This will render a completely new page, but with the same content. Now if you find yourself in a situation where this is the case you’ve got one mighty ally which is the redirect. You can even create a redirect-rule which will help you do this in bulk. This will redirect all the http-pages to https, and all the pages with a trailing slash to the ones without a trailing slash. Again this will take time for all the pages to be removed from the index. But as you have a redirect in place it will also never be indexed anymore.

Transitioning to a New Domain

I could write a separate blog post about this, as a domain transition requires following a meticulous process to ensure all gained rankings are preserved. Most important thing when it comes to indexing is that the old-urls are redirected to their new counterparts. This can be a one-to-one relation which will ensure the best chance of preserving a ranking for a page. It can be done in bulk on a directory level or even on the domain level. Both those options will not preserve rankings and will also detract to the user experience. A user will need to do the search after it lands on the new website. Best thing to remember here is that you need a redirect plan as specific as you can make it.

If you, for whatever reason, don’t want to preserve any rankings and want to completely remove a domain from the index, you now know that you’ve got several options at your disposal.

• You could delete the entire website and let Google clean up any of the 404s it encounters.
• Or you could set the whole domain on noindex and let Google remove all the URLs from the index.

But again don’t restrain indexing from the robots.txt file as Google won’t be able to see your pages any more. Leaving your old domain in the index.

Verifying De-indexing

Google Search Console, as the name implies, is your go to when it comes to checking your websites indexation. Here you can get an overview of which pages are indexed and which are not.

The latter all grouped under different reasons. For example because of the noindex directive.

There is another way you can check if a page is still available in the index: by Googling your page. To be more specific with the site: or inUrl: directives you can narrow down your search.

site:https://acceptance.yourwebsite.com

This will show you all the pages for that particular site or page. If it still shows up in the search results that means: Google either hasn’t visited the page yet or your setup is not correct enabling Google to still index the page. If it is just one page, you can also request a re-indexation of the page through Google Search Console. This can speed up the process, but can only be done per page. So sometimes you just need to play the waiting game.

Considerations and Consequences

Coming to the end of this article, there is one more important thing to consider. This has been mentioned a couple of times, but I like to dedicate some special attention to it. De-indexing pages will have an effect on the traffic to your website. If you remove a page from Google's index, it will not be able to show that page anymore to its users. So be careful when you start removing pages.

If the reason for removal is abundantly clear (e.g. the page shouldn’t be publicly available) there should be no worries. But when it comes to outdated content, page- or website-migrations it is important to have a good understanding of what you’re trying to do and achieve. Consider hiring a professional to help when you find yourself at such a crossroad. Because it can take a long time to get pages to rank high in the results, but you can lose them very fast.