Please tell me the following: abstract situation - in the robots.txt file indexing of files \ directories is completely prohibited, and individual pages are listed in the sitemap.xml file. What will the search robot do? Which file is more priority? I will explain the cause of the issue, there is an online store (+ - 5000 products), it works on a samopisny engine (PHP + MySQL). Added autogeneration sitemap, now you need to align the contents of the map with the file robots.txt.

  • in the robots.txt file, specify the parameter path to sitemap.xml; sitemap: http://example.com/site_structure/my_sitemaps1.xml - tCode

2 answers 2

The robots.txt file will take priority, at least on Google, these are their words :

If you’re still interested, you’ll still be able to find the correct URL.

  • As I understand it, Google can simply index pages that are prohibited for indexing on the "native" site, but links to which are contained on other sites. - Vyacheslav Kirichenko

Help Google reports: Robots.txt instructions are advisory in nature . If you want to close some pages from indexing, then use noindex - info Google . I think sitemap is more important for indexing.