How to request Python 3 via an HTTP proxy

Question

In Python, we can recover the contents of a URL in a very simple way:

from urllib import request
response = request.urlopen('http://www.google.com')

However, in corporate environments we cannot access the internet directly, we need to go through a Proxy. How to do this?

Answer 1

Python 3 solution using urllib

You can create a request using a opener which, in turn, receives information from a Proxyhandler:

from urllib import request

#configura um "opener" com um proxy
proxy  = request.ProxyHandler({"http":"proxy:8080"})
auth = request.HTTPBasicAuthHandler()
opener = request.build_opener(proxy, auth, request.HTTPHandler)

#faz uma requisição usando o "opener"
response = opener.open('http://www.google.com')
print(response.read().decode('ISO-8859-1'))

It is also possible to define a global proxy configuration as follows:

from urllib import request

#configura um "opener" com um proxy
proxy  = request.ProxyHandler({"http":"proxy:8080"})
auth = request.HTTPBasicAuthHandler()
opener = request.build_opener(proxy, auth, request.HTTPHandler)

#instala o "opener" globalmente
request.install_opener(opener)

#faz uma requisição genérica (sem o opener)
response = request.urlopen('http://www.google.com')
print(response.read().decode('ISO-8859-1'))

Python 2 solution using urllib2

Changes a little bit in relation to Python 3, remembering that the urllib2 version 2 is "equivalent" to urllib version 3, although functions and classes have changed location and name:

import urllib2

#proxy
proxy = urllib2.ProxyHandler({"http":"proxy:8080"})
opener = urllib2.build_opener(proxy)
urllib2.install_opener(opener)

response = urllib2.urlopen("http://www.google.com")
print response.read()

Solution using the library requests

To add the proxy to a simple request:

import requests

response = requests.get("http://www.google.com",
                        proxies = { "http": "proxy:8080" })
print(response.text)

To add the proxy in multiple requests:

import requests

sessao = requests.Session()
sessao.proxies = {"http": "proxy:8080"}

response = sessao.get("http://www.google.com")
print(response.text)

Note: Remembering that this is a separate library installed.

---

General note: Examples are for an HTTP proxy that does not require user and password.

Answer 2

Just set the proxy you want to use via an environment variable. In Windows the proxy configuration is obtained directly from the internet explorer configuration.

But it also depends on the type of proxy. If it’s something like MS proxy server using NTLM for authorization, then you’re gonna have to use something like that:

http://code.google.com/p/python-ntlm/