In requests, how to correctly read the ISO-8859-1 encoding?

Question

In Python3, with beautifulsoup4 and requests, I want to extract some information from a site that has encoding 'ISO-8859-1'. I tried this strategy to show correctly the text:

import requests
from bs4 import BeautifulSoup

req = requests.get('https://sisgvarmazenamento.blob.core.windows.net/prd/PublicacaoPortal/Arquivos/201901.htm')
req.encoding

encoding = req.encoding
text = req.content

decoded_text = text.decode(encoding)

sopa = BeautifulSoup(decoded_text, "lxml")

sopa.find("h1")

And the result that appears is:

<h1>
                        CÃMARA MUNICIPAL DE SÃO PAULO<br/></h1>

When I copy and paste this screen appears correct, but in my computer the whole accent is wrong

I’m on a machine with Ubuntu

Please someone knows a correct way to read the encoding?

---

Edited June 2, 2019

I had the help of @snakecharmerb here

In the answer he detailed that when no explicit charset is present in HTTP headers and the Content-Type header contains text, RFC 2616 specifies that the standard character set should be ISO-8859-1. What is the case with this site

But clearly the words are UTF-8, so I fix it manually and it works My code went like this and it worked:

import requests
from bs4 import BeautifulSoup

req = requests.get('https://sisgvarmazenamento.blob.core.windows.net/prd/PublicacaoPortal/Arquivos/201901.htm')
req.encoding

'ISO-8859-1'

req.headers['content-type']
'text/html'

req.encoding = 'UTF-8'

sopa = BeautifulSoup(req.text,'lxml')

sopa.find('h1').text
'\r\n                        CÂMARA MUNICIPAL DE SÃO PAULO'

Answer 1

In fact, the requests already makes the decoding for you,using the correct encoding.

It’s only instead of accessing the attribute .content, access the attribute .text of the object "":

In [382]: import requests                                                                                              

In [383]: data = requests.get("https://slashdot.org")                                                                  

In [384]: type(data.content)                                                                                           
Out[384]: bytes

In [385]: type(data.text)                                                                                              
Out[385]: str

But understanding of encodnig and not kicking what happens is kind vital in this industry. I never tire of recommending the following article, originally written in 2003 by the creator of Stackoverflow:

The Absolute Minimum All Software Programmers Need, Absolutely, Positively To Know About Unicode and Character Sets (No Excuses!)

Answer 2

If you refer to display in terminal or windows command prompt then to resolve you should set in python the charset script, thus:

# -*- coding: latin-1 -*-

import requests
from bs4 import BeautifulSoup

...

If the page is in iso-8859-1 or windows-1252, if it is in utf-8 use so:

# -*- coding: utf-8 -*-

import requests
from bs4 import BeautifulSoup

...

It seems to me that the page that linked this in utf-16, so I think it would be like this:

# -*- coding: utf-16 -*-

import requests
from bs4 import BeautifulSoup

...