Yuzayu one of the ways to get data from this site:

import mechanize br = mechanize.Browser() br.open("http://www.example.com/")

well or requests if cookies are not needed. Is it possible to do the same using any type of proxy 111.111.111.111:8080?

  • Here are some examples of working with http and SOCKS5 proxies using urrlib. - jfs
  • In Python 3, urllib3 is used, do not tell me where you can read about the differences and generally learn how to use it correctly? If it is in Russian, it is generally excellent. The task is to use proxies and cookies (ideally to emulate a browser), mechanize disappears, because he is not friendly with Python 3. - Oleg
  • urllib3 is not related to Python 3 — it is an independent library that works on both Python 2 and 3. The examples above are shown for the standard library (one for Python 2, the other for Python 3) —API with the accuracy of imports remained the same (build_opener () and urlopen ()). - jfs

1 answer 1

The mechanize browser (works in Python 2) has a method for setting a proxy - set_proxies :

 #!/usr/bin/env python2 import mechanize br = mechanize.Browser() proxies = { "http": "111.111.111.111:8080", "ftp": "proxy.example.com", } br.set_proxies(proxies) br.open("http://www.example.com/")

There are helpful examples on the docks.

Through a proxy, you can also go to requests (works in Python 2 and 3):

 #!/usr/bin/env python import requests proxies = { 'http': '10.10.1.10:3128', 'https': '10.10.1.10:1080', } requests.get('http://example.org', proxies=proxies)

SOCKS proxies are also supported. Details in the documentation .

And yes, requests course can work with cookie . RTFM, as they say!

  • Unfortunately we could not make friends mechanize with Python 3. - Oleg
  • @ Oleg means make friends with requests - a library for all occasions. Updated the answer. - zed
  • @zed: to indicate on which versions your examples can be run and that they can work as separate scripts, you can add shebang ( #! line) . For example, #!/usr/bin/env python2 for the first example and #!/usr/bin/env python for the second (the second example both on Python 2 and 3 should work). - jfs
  • Could you explain what you have invested in the concept of “labeling” because in this context it can be interpreted in different ways? - Oleg
  • @jfs Thanks, added the answer. - zed