What is WordPress Robots.txt - In-depth Tutorial

What is WordPress Robots.txt - In-depth Tutorial


The topics we will cover in this tutorial are as follows:


What is robots.txt?


Where to find WordPress robots.txt


How to create a robots.txt file


How should WordPress robots.txt be


What is WordPress robots.txt ( and why it is required )


wordpress-robottxt-1.PNG

When you create a new website, search engines crawl by sending their bots and create a map of all the pages it contains. In this way, when someone searches for a related keyword, they determine which pages to show as a result.


The main problem is that modern websites contain a lot more elements than pages. WordPress, for example, lets you install plugins that usually have their own directories. You don't want them to appear in search results because they are not related to your content.


The job of the WordPress robots.txt file is to give a set of instructions to search engines. It actually tells them: "You can look here, but don't enter into other rooms beyond!" This file can be as detailed as you want and is very easy to create.


In practice, search engines will still crawl your site even if you haven't created a robots.txt file. However, not to create this file will result in inefficiency. Without this file, you leave all the work to the bots to index your content. Other people are likely to see pages on your website that they don't have access to as search results.


More importantly, when there is no robots.txt file, there will be too many bots crawling your website. This situation negatively affects performance. Page loading speed should always be your priority. As a matter of fact, people hate slow websites.


WordPress robots.txt File Location


When you create a WordPress site, the system automatically creates a virtual robots.txt file in your server's home folder. For example, if your site is located at samplewebsite.com, you should be able to open samplewebsite.com/robots.txt and see a file like the following:


User-agent: *
Disallow: /wp-admin/
Disallow: /wp-includes/

The WordPress robots.txt settings in this example are pretty simple. The asterisk(*) means the rules are universal and will apply to all bots. In our example, our file tells bots that they cannot enter the wp-admin and wp-includes directories. These two folders contain very sensitive information.


However, you may want to add more rules to your own file. Before getting started, you need to understand that this is a virtual file. Usually the WordPress robots.txt file location is the same as the root directory and is often referred to as public_html or www (or your website name):


wordpress-robottxt-2.PNG

However, the default WordPress robots.txt file is not accessible from any directory. It works, but since you can make changes, you have to create your own file and replace it by uploading it to the root folder.


WordPress robots.txt How To Be And What Rules Should It Contain


You saw a sample robots.txt file created by WordPress above. There are only two short rules in this example, but most websites add much more than that. Let's compare two different robots.txt files and see their differences.


Here's our first WordPress robots.txt example:

User-agent: *
Allow: /
# Disallowed Sub-Directories
Disallow: /checkout/
Disallow: /images/
Disallow: /forum/

You see a section like Allow: /. This line tells bots that they can crawl your website's pages but ignore the exceptions you set


Let's take a look at another WordPress robots.txt file:


User-agent: *
Disallow: /wp-admin/
Disallow: /wp-includes/
User-agent: Bingbot
Disallow: /

In this file, we entered the same rules that WordPress set as the default. However, we also added a new rule that does not allow Bing's search bot to crawl your site. Bingbot is the name of this bot.


It is important to remember that although most bots follow the instructions you set in this file, you are not forcing them to do so, just requesting nicely.


Here is an example of how the WordPress robots.txt should be:


User-Agent: *
Allow: /wp-content/uploads/
Disallow: /wp-content/plugins/

Traditionally, WordPress likes to block access to the wp-admin and wp-includes directories. However, this is no longer seen as a useful approach. Plus, if you're adding metadata to your images for search engine optimization (SEO), there is no logical explanation for preventing bots from accessing this information. Instead, the above two rules will meet the needs of all basic sites.


How to Create a WordPress robots.txt File Using Yoast SEO


No need to explain the Yoast SEO plugin. WordPress's most popular SEO plugin and allows you to optimize your articles and pages to better reflect keywords. Apart from this, it helps to increase the readability of your content and thus your visitors can have more enjoyable time.


After installing and activating the plugin, open the SEO > Tools section from the admin panel and click on File Editor.

wordpress-robottxt-3.PNG

wordpress-robottxt-4.PNG

Clicking this link will take you to a new page where you can edit the .htaccess file. You can also click the Create robots.txt file button.


wordpress-robottxt-5.PNG

When you click this button, you will see a new editor in the tab, from which you can directly edit the robots.txt file. Remember, Yoast SEO sets its own default rules, which will replace the existing virtual robots.txt file.


Whenever you add or remove a new rule, don't forget to click the "Save changes to robots.txt" button:


wordpress-robottxt-6.PNG

Creating and Uploading WordPress robots.txt Via FTP


Creating a robots.txt file is one of the simplest jobs. All you have to do is open any text editor (like Notepad) and type a few lines. You may want to create the WordPress robots.txt file this way without using a plugin.


As an example file we can show:


wordpress-robottxt-7.PNG

We have saved this file directly on our own computer. Once you have created and saved your file, you need to connect to your website via FTP. We use the FileZilla FTP program


Open the public_html folder when you connect to your site. Then all you have to do is upload the robots.txt file on your computer to your server. For this, you can right-click the file on the FTP program or drag and drop the file to the relevant area:


wordpress-robottxt-8.PNG