Hello friends admins. Brainstorming needed. There are several sites with a third-level domain on my domain, these are test sites for clients. The task is this: to restrict access so that only certain people can get to a specific site, while sites are not indexed anywhere in search engines and not displayed.

What are the suggestions in the implementation?

PS is used only NGINX.

    2 answers 2

    Authorization through mod_auth_digest

    • Oh, cool. thank. it would fit. Is there a good manual to learn? - Bansh
    • Found, implemented. thank you very much. Here is a great manual - Bansh

    So what exactly needs to be done? To ban indexing or to make access only to certain people?

    If you just want to prevent robots from indexing, then you can use the robots.txt file located in the root of the site with the following contents:

     User-agent: * Disallow: /

    If you want to do this through nginx - then you can make restrictions on the identifier:

     if ($http_user_agent ~* (перечень|user|agent) ) { return 403; }

    If you need to allow access to the site only from certain addresses, then do so:

     server { listen ...; server_name ...; allow 1.1.1.1; allow 9.8.7.6; allow 73.23.41.98; deny all; ... }
    • robots.txt GOOGLE ignores. that's the problem ... - Bansh
    • @AnanivSergey: why GOOGLE can index the page despite the robots.txt file described in the Google documentation - support.google.com/webmasters/answer/6062608?hl=en . To avoid this, you must use the robots meta tag or the X-Robots-Tag header. For example - <meta name="robots" content="noindex" /> . Examples and meanings here - developers.google.com/webmasters/control-crawl-index/docs/… - MANKK
    • Yes. I know but this is not convenient, because when the project leaves the test stage, all these tags will have to be deleted. - Bansh