What is a robots.txt file? What is the impact of this on search engine optimization?

To better understand the meaning and use of bots. txt file, you should first learn about SEO. In fact, “search engine optimization” is all the actions we do on our website so that its pages can be seen in Google search results. For example, after searching for the word buy Xiaomi mobile, you will see Digikala in the first link of Google. why?

This is because this website has given a lot of importance to its SEO and Google has noticed its high reputation. In this way, this website page appears to the public as a result of searching for the desired word more than other sites. Now we can easily understand what robots.txt is and why it is used?!

What is a robots.txt file?

The robots.txt file is a text file on our site that contains a number of commands for commanding robots and web crawlers! Let me explain more simply.

As you can see, the primary step that Google takes to rank the pages of our site in the search results is crawling the pages. what does that mean? In fact, in this step, Google visits our website with the help of robots known as Googlebot and checks our pages and contents one by one. Then, it saves (indexes) these pages and gives each one a rank in the relevant words in the search results according to the SEO factors. For example, the “Improve SEO” page on the portal goes to page 5 of the search results.

Now, our duty as a webmaster or SEO is to take every measure so that Google robots can check our site more easily and not go to any worthless page.

One such procedure is optimizing your robots.txt file. In fact, the robots.txt file is just a text file in which a number of English scripts similar to programming code are written. Given that robots do not understand our human language, we can give necessary commands to robots with the help of these codes. For example, tell them not to check a certain page of our site or not to visit our site at all for a period of time because the site is under repair.

Is the robots.txt file just for telling robots?

Somehow, you can say yes! In fact, the codes in this file are intended to give instructions to Google robots and other search engines. Commands such as the one that does not check a specific page or, for example, does not display images of the entire site or part of the pages to users in the search results.

But there is an important point. If there is wrong code in the file, the SEO of our site may be lost. Why?! As I said above, the first step for Google to display site pages in search results is to check our pages. Now, when we accidentally or intentionally do something with code in robots.txt, these robots don’t check the pages of the site, we’ve practically blocked our site’s SEO!

Do we need to create/modify a robots.txt file?

In this article I want to explain to you everything you need to know about bots as a webmaster. But before going to the expert discussions, as someone with at least 5 years of content creation and SEO site experience, I have to make this important point:

Working with robots txt is not very easy and even some SEOs don’t know how to handle it. For this reason, you should not go for it as a user without experience and expertise.

The most important point here is that if you build your website using popular CMS; For example, a WordPress system or a portal site builder, in this case your file is fully optimized and does not need any changes. So try not to go directly to him. In particular, you can apply various commands in this file with the help of newindex and index tags, which are easily available in your CMS. (Read the article on meta tags.)

But if you have a very unreliable CMS or at least your site was designed for you by someone who is not familiar with SEO, you may have to go through this file to make some minor changes in the future.

What is the importance and use of robots.txt?

Well, I’ve given as much explanation for beginners as necessary and it’s best to move on to the specialized part of the article from here on. Next, I will explain to you that you can use robots.txt on the site?

1_ Block pages

Especially if you have entered the world of SEO professionally, you will understand that very often on the site you have to hide pages from robots! For example, a page you designed for users, but its content is less than 100 words, or a page with duplicate content compared to one of your site’s pages or other people’s site pages.

In fact, in this case, if the Google bot sees the requested pages, it may reduce the overall SEO power of your site due to their inappropriate content. Therefore, the best option is to hide them from their view and instruct bots not to check the page in question.

Anyway, one of the uses of robots.txt is that you can tell robots not to scan the page or pages you want. Of course, be warned that most site building systems, including Portal Site Builder and WordPress, allow you to easily add a Nuindex tag to a page with the help of a button on the desired page and remove it from bots. The text becomes superfluous.

2_ Help improve the creeping budget

This text is generally placed at the beginning of the page, and it is in it that the exact address of the sitemap should be specified. This time, to specify the address, you must type it in full. It means something similar to the text below:


As is known, the title of the sitemap page is written in full. After creating the file and placing the required commands in it, save it as robots.txt.

Now enter Hosting Spaces. Upload the file to the host and to the root root (the main folder of the host where the site information is located). If you did it right, then after entering the site address and adding /robots.txt to the end of the domain, you will see the text file you created. That is, you must enter this address in the browser to see the file:

Note: Before you finish this part of the article, you should know two more things about creating a robots.txt file:

  •      The / command, unlike the above commands, means that all requested pages include the command. For example, Disallow: / means that not all pages can be seen.
  •      You can also put an exact page address in front of the commands. For example, Disallow: /blog/business means that this page should not be viewed on the site.

Android sample file

After several sites to find a good robots.txt example, I finally came to moz’s robots:

As you can see, there are many commands in this file. But all of them are finally one of the four orders that we met above. If you want to see this file for yourself, you can enter this address in your computer’s browser:

The last word

Well, we have come to the end of the article. In this article, we first understood what a robots txt file is and what it is used for. Also, in the second part, we learned how to deal with creating a robots.txt file. Finally, if you have any questions or comments, I am waiting for you in the comments section.