The site is running CNC, but Google is still crawling to scan links like /index.php?variable=value. From where he hawkes these links even I can not put my mind to it. Put the header location in case there is a "?" In the URL. Redirection works to this day, but this is not good for the site, because search engine climbs to index broken links instead of indexing what they say to him ...

In short, closer to the point. How to make a correct ban on the indexation of all that is after. Php?

Will the next solution be correct?

Disallow: /index.php *

    2 answers 2

    I somehow did something like:

    Disallow: /profile.php?username=*

    It worked correctly, as far as I remember :)

    • I will try, but I generally need to cut everything down after .php - Palmervan
    • In general, as far as I remember in robots.txt, full regulars should be supported. - Shamanis
    • While I’ll try with my example, I’ll send it now, I’ll see the bot shavat new robots. Thank. - Palmervan
     User-agent: * Disallow: /*?твоя переменная* ну или Disallow: /*.php // запретить индексацию всех php файлов встреченных в url