There is a one-page WordPress site. It is a loaded and activated theme, that is, there are no registered widgets, sidebars, etc. The only thing that comes from the admin is the name and description of the site. All media files are also in the theme folder.

There are questions - how in this case to register robots.txt? What folders to prohibit / allow for indexing? Sample robots.txt from the Internet for this situation is clearly not suitable, since all of them offer to prohibit indexing folder wp-themes, and the folder wp-includes, on the contrary, allow.

Should the robots.txt data be rewritten in this situation by forbidding the indexing of wp-includes and enabling the indexing of wp-themes? And is it possible to simply prohibit everything except the folder with the theme for indexing ?!

  • I see no problem: the bot indexes what there are links to. if you have a small one-page, then the number of links, especially internal ones, tends to zero. It seems to me that you should not think so seriously about this at all. - Ivan Pshenitsyn

1 answer 1

Detailed robots.txt file

User-agent: * Disallow: /cgi-bin # классика... Disallow: /? # всС ΠΏΠ°Ρ€Π°ΠΌΠ΅Ρ‚Ρ€Ρ‹ запроса Π½Π° Π³Π»Π°Π²Π½ΠΎΠΉ Disallow: /wp- # всС Ρ„Π°ΠΉΠ»Ρ‹ WP: /wp-json/, /wp-includes, /wp-content/plugins Disallow: /xmlrpc.php Disallow: *?s= # поиск Disallow: *&s= # поиск Disallow: /search # поиск Disallow: /author/ # Π°Ρ€Ρ…ΠΈΠ² Π°Π²Ρ‚ΠΎΡ€Π° Disallow: *?attachment_id= Disallow: */feed Disallow: */page/ # всС Π²ΠΈΠ΄Ρ‹ ΠΏΠ°Π³ΠΈΠ½Π°Ρ†ΠΈΠΈ Allow: */uploads Allow: /*/*.js # Π²Π½ΡƒΡ‚Ρ€ΠΈ /wp- (/*/ - для ΠΏΡ€ΠΈΠΎΡ€ΠΈΡ‚Π΅Ρ‚Π°) Allow: /*/*.css # Π²Π½ΡƒΡ‚Ρ€ΠΈ /wp- (/*/ - для ΠΏΡ€ΠΈΠΎΡ€ΠΈΡ‚Π΅Ρ‚Π°) Allow: /wp-*.png # ΠΊΠ°Ρ€Ρ‚ΠΈΠ½ΠΊΠΈ Π² ΠΏΠ»Π°Π³ΠΈΠ½Π°Ρ…, cache ΠΏΠ°ΠΏΠΊΠ΅ ΠΈ Ρ‚.Π΄. Allow: /wp-*.jpg # ΠΊΠ°Ρ€Ρ‚ΠΈΠ½ΠΊΠΈ Π² ΠΏΠ»Π°Π³ΠΈΠ½Π°Ρ…, cache ΠΏΠ°ΠΏΠΊΠ΅ ΠΈ Ρ‚.Π΄. Allow: /wp-*.jpeg # ΠΊΠ°Ρ€Ρ‚ΠΈΠ½ΠΊΠΈ Π² ΠΏΠ»Π°Π³ΠΈΠ½Π°Ρ…, cache ΠΏΠ°ΠΏΠΊΠ΅ ΠΈ Ρ‚.Π΄. Allow: /wp-*.gif # ΠΊΠ°Ρ€Ρ‚ΠΈΠ½ΠΊΠΈ Π² ΠΏΠ»Π°Π³ΠΈΠ½Π°Ρ…, cache ΠΏΠ°ΠΏΠΊΠ΅ ΠΈ Ρ‚.Π΄. #Disallow: /wp/ # ΠΊΠΎΠ³Π΄Π° WP установлСн Π² ΠΏΠΎΠ΄ΠΊΠ°Ρ‚Π°Π»ΠΎΠ³ wp Host: www.site.ru Sitemap: http://site.ru/sitemap.xml Sitemap: http://site.ru/sitemap2.xml

As you can see, indexing / wp-content / themes / is not prohibited. What is required.

But usually everything is made much easier. Here is a robots.txt file generated by Yoast SEO (about a million installations)

 User-agent: * Disallow: /wp-admin/ Allow: /wp-admin/admin-ajax.php

Only one thing is important - to block access to / wp-admin and allow ajax. All the rest is lyrics.