Regular expression (regex) for links to web pages using Python

Question

I am trying to learn how to create a webcrawler. Part of the code will be to extract links on a web page (links starting with http or https):

import re   
urls = re.findall(r'href=[\'"]?([^\'" >]+)', s)

How can I modify or create a new rgex that just picks up links that start with http or https? I don’t want to keep the word "href" just "http://..." or "https://..." They do not serve, for example: "media/test", "G1/noticia"

padrao = re.findall(r'href=[\'"]https?://[\w:/\.\'"_]+' ,html)

standard also not 100% functional:

http://g1.globo.com/rio

http://g1.globo.com/"

Some left with " at the end, which was not meant to occur!

Answer 1

To complete the excellent @zekk response, here’s a solution for python 3.x:

import requests, re

url = "/q/143677"
html = requests.get(url).text

urls = re.findall('(?<=href=["\'])https?://.+?(?=["\'])', html)
print(urls)

Answer 2

import urllib, re

url = "/q/143677"
html = urllib.urlopen(url).read()

urls = re.findall('(?<=href=["\'])https?://.+?(?=["\'])', html)

for url in urls:
    print url

Regular expression will match anything between href=" and " and may it be http or https.

You can also make use of a parser for this, for example the Beautiful Soup:

import urllib, re
# Para a versao 4.x use from bs4 import BeautifulSoup
from BeautifulSoup import BeautifulSoup

url = "/q/143677"
html = urllib.urlopen(url).read()

soup = BeautifulSoup(html)
urls = soup.findAll('a', attrs={'href': re.compile("^https?://")})

for tag in urls:
    print tag['href']