I am writing a parsing html page in python, after several requests the site is not available - it simply does not open (I go through a proxy - everything works). The question is whether it is necessary to prescribe something in response so that the server does not define me as a "malicious parser"? ))

  • Sites block access usually after numerous requests, then using a proxy can help. And what site are you parsing, if not a secret? :) - gil9red
  • Maybe first on the site itself to wander? Suddenly there is something like “policies for bots”, or in general - which API or necessary RSS feeds will be found ... - PinkTux
  • gil9red, shopindia.ru - this site. I don’t even understand why it’s like that I don't do a hundred requests at a time - Ruben
  • one
    This is probably a ban. So that the server does not identify you as a malicious parser, stop it being. ) - Nick Volynkin ♦
  • one
    The point may not be in the number of requests, but in their frequency. Plus one IP, plus UserAgent - that's the ban. - PinkTux

0